[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:55:50 dist-test-slave-fl7g krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:55:50 dist-test-slave-fl7g krb5kdc[399](info): set up 2 sockets
Sep 26 20:55:50 dist-test-slave-fl7g krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 20:55:51.853539 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:55:51 dist-test-slave-fl7g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:55:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:55:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:55:51.979805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37011
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:52.063135 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:52.063375 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:52.063400 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:52.064601 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:52.064646 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:52.064661 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:52.064673 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:52.064685 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:52.066193 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:52.066417 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:52.066644 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:55:52.072687 412 server_base.cc:1034] running on GCE node
W20260926 20:55:52.072867 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:52.073134 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:52.073812 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:52.074116 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:52.074481 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:52.075664 412 hybrid_clock.cc:648] HybridClock initialized: now 1790456152075645 us; error 37 us; skew 500 ppm
Sep 26 20:55:52 dist-test-slave-fl7g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:52.078296 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:55:52.078792 412 webserver.cc:460] Webserver started at http://127.0.98.62:40885/ using document root <none> and password file <none>
I20260926 20:55:52.079010 412 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:52.079053 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:52.079149 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:52.080214 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "ed7a32e1824045ddb104621e193eb809"
format_stamp: "Formatted at 2026-09-26 20:55:52 on dist-test-slave-fl7g"
server_key: "ef9ca689e238d231b475fdcd3816a15a"
server_key_iv: "89e78bc993ec3ae0e07d806b7ec39c98"
server_key_version: "encryptionkey@0"
I20260926 20:55:52.080621 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed7a32e1824045ddb104621e193eb809"
format_stamp: "Formatted at 2026-09-26 20:55:52 on dist-test-slave-fl7g"
server_key: "ef9ca689e238d231b475fdcd3816a15a"
server_key_iv: "89e78bc993ec3ae0e07d806b7ec39c98"
server_key_version: "encryptionkey@0"
I20260926 20:55:52.082108 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:55:52.082943 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:52.083161 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:55:52.083258 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "ed7a32e1824045ddb104621e193eb809"
format_stamp: "Formatted at 2026-09-26 20:55:52 on dist-test-slave-fl7g"
server_key: "ef9ca689e238d231b475fdcd3816a15a"
server_key_iv: "89e78bc993ec3ae0e07d806b7ec39c98"
server_key_version: "encryptionkey@0"
I20260926 20:55:52.083326 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:52.106892 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:52.109030 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:52.109164 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:52.113127 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37011
I20260926 20:55:52.113165 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37011 every 8 connection(s)
I20260926 20:55:52.113588 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:55:52.114557 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:52.115360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 412
I20260926 20:55:52.115473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:55:52.116310 392 external_mini_cluster.cc:1428] Setting key c5b68ca3c812f81b9e5fd7e7123c8b70
I20260926 20:55:52.117720 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809: Bootstrap starting.
I20260926 20:55:52.118387 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:52.118696 480 log.cc:824] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:52.119524 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809: No bootstrap required, opened a new log
I20260926 20:55:52.121555 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7a32e1824045ddb104621e193eb809" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260926 20:55:52.121729 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:52.121801 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7a32e1824045ddb104621e193eb809, State: Initialized, Role: FOLLOWER
I20260926 20:55:52.121976 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7a32e1824045ddb104621e193eb809" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260926 20:55:52.122073 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:55:52.122120 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:55:52.122169 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:52.123255 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7a32e1824045ddb104621e193eb809" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260926 20:55:52.123368 480 leader_election.cc:302] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed7a32e1824045ddb104621e193eb809; no voters:
I20260926 20:55:52.123543 480 leader_election.cc:288] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:55:52.123637 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:52.123795 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:52.123844 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809 [term 1 LEADER]: Becoming Leader. State: Replica: ed7a32e1824045ddb104621e193eb809, State: Running, Role: LEADER
I20260926 20:55:52.124018 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7a32e1824045ddb104621e193eb809" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260926 20:55:52.124395 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed7a32e1824045ddb104621e193eb809. Latest consensus state: current_term: 1 leader_uuid: "ed7a32e1824045ddb104621e193eb809" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7a32e1824045ddb104621e193eb809" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } }
I20260926 20:55:52.124691 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809 [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:52.124625 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed7a32e1824045ddb104621e193eb809" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7a32e1824045ddb104621e193eb809" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } }
I20260926 20:55:52.125063 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809 [sys.catalog]: This master's current role is: LEADER
W20260926 20:55:52.125816 501 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:55:52.125872 501 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:55:52.125931 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:55:52.126107 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:55:52.127645 494 catalog_manager.cc:1347] Generated new cluster ID: a432ce1c5ee64e2da21f6a9fa3d81b3e
I20260926 20:55:52.127700 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:55:52 dist-test-slave-fl7g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:55:52.135885 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:52.117429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47092 (local address 127.0.98.62:37011)
0926 20:55:52.117634 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:52.117643 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:52.117985 (+ 342us) server_negotiation.cc:407] Connection header received
0926 20:55:52.120783 (+ 2798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:52.120793 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:52.120858 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:52.121014 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:52.122783 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:52.124214 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:52.125835 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:52.126390 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:52.126548 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:52.131052 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:52.131057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:52.131057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:52.131071 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:52.133375 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:52.133979 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:52.133983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:52.133988 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:52.134120 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:52.134482 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:52.134483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:52.134483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:52.134813 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:52.134983 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:52.135303 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 20:55:52.135455 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":90,"threads_started":1}
I20260926 20:55:52.150995 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:55:52.151568 494 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:55:52.154160 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed7a32e1824045ddb104621e193eb809: Generated new TSK 0
I20260926 20:55:52.154408 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:55:52.184384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:52.270750 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:52.271006 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:52.271054 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:52.272284 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:52.272364 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:52.272410 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:55:52.274034 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:52.274329 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:52.274588 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:52.280273 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:52.280315 506 server_base.cc:1034] running on GCE node
W20260926 20:55:52.280251 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:52.280253 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:52.280834 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:52.281139 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:52.282325 506 hybrid_clock.cc:648] HybridClock initialized: now 1790456152282311 us; error 33 us; skew 500 ppm
Sep 26 20:55:52 dist-test-slave-fl7g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:52.285036 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:55:52.285614 506 webserver.cc:460] Webserver started at http://127.0.98.1:46269/ using document root <none> and password file <none>
I20260926 20:55:52.285861 506 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:52.285933 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:52.286091 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:52.287302 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "2f0b7bb8e14d418585f667b6f39eb4de"
format_stamp: "Formatted at 2026-09-26 20:55:52 on dist-test-slave-fl7g"
server_key: "7a58cb2d826f23af234f10a64a1ba07f"
server_key_iv: "7a8208952ac0c4366d6d615755ba96d6"
server_key_version: "encryptionkey@0"
I20260926 20:55:52.287747 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2f0b7bb8e14d418585f667b6f39eb4de"
format_stamp: "Formatted at 2026-09-26 20:55:52 on dist-test-slave-fl7g"
server_key: "7a58cb2d826f23af234f10a64a1ba07f"
server_key_iv: "7a8208952ac0c4366d6d615755ba96d6"
server_key_version: "encryptionkey@0"
I20260926 20:55:52.289366 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:55:52.290233 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:52.290470 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:55:52.290585 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "2f0b7bb8e14d418585f667b6f39eb4de"
format_stamp: "Formatted at 2026-09-26 20:55:52 on dist-test-slave-fl7g"
server_key: "7a58cb2d826f23af234f10a64a1ba07f"
server_key_iv: "7a8208952ac0c4366d6d615755ba96d6"
server_key_version: "encryptionkey@0"
I20260926 20:55:52.290704 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:52.300545 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:52.303269 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:52.303459 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:52.303750 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:52.304080 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:52.304133 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:52.304169 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:52.304214 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:52.310406 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38963
I20260926 20:55:52.310492 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38963 every 8 connection(s)
I20260926 20:55:52.310837 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 20:55:52 dist-test-slave-fl7g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:55:52.320111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 506
I20260926 20:55:52.320241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:55:52.320452 392 external_mini_cluster.cc:1428] Setting key 5072e107a845098509653a8c60318a55
I20260926 20:55:52.321039 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:52.311735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60015 (local address 127.0.98.62:37011)
0926 20:55:52.311820 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:52.311822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:52.311883 (+ 61us) server_negotiation.cc:407] Connection header received
0926 20:55:52.312389 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:52.312391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:52.312400 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:52.312557 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:52.313672 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:52.314195 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:52.315232 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:52.315451 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:52.315540 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:52.317956 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:52.317957 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:52.317957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:52.317965 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:52.319330 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:52.319831 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:52.319832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:52.319832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:52.319887 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:52.320194 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:52.320195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:52.320196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:52.320357 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:52.320465 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:52.320897 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 20:55:52.320974 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:55:52.321347 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:52.311592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37011 (local address 127.0.98.1:60015)
0926 20:55:52.311765 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:52.311809 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:52.312294 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:52.312665 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:52.312669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:52.313333 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:52.313541 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:52.313546 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:52.314291 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:52.314292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:52.315112 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:52.315114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:52.315534 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:52.315535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:52.315639 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:52.316213 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:52.316222 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:52.317850 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:52.319429 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:52.319438 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:52.319440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:52.319749 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:52.319975 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:52.319976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:52.319976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:52.320110 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:52.320464 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:52.320478 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:52.320729 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 20:55:52.320883 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 20:55:52.321073 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260926 20:55:52.321893 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37011
I20260926 20:55:52.322016 635 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:52.322217 635 heartbeater.cc:511] Master 127.0.98.62:37011 requested a full tablet report, sending...
I20260926 20:55:52.322857 444 ts_manager.cc:194] Registered new tserver with Master: 2f0b7bb8e14d418585f667b6f39eb4de (127.0.98.1:38963)
I20260926 20:55:52.323853 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60015
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:55:52.348829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:52.433091 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:52.433362 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:52.433405 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:52.434671 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:52.434753 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:52.434803 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:55:52.436784 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:52.437141 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:52.437422 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:52.443248 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:52.443359 640 server_base.cc:1034] running on GCE node
W20260926 20:55:52.443248 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:52.443243 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:52.443827 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:52.444113 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:52.445288 640 hybrid_clock.cc:648] HybridClock initialized: now 1790456152445270 us; error 27 us; skew 500 ppm
Sep 26 20:55:52 dist-test-slave-fl7g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:52.447816 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:55:52.448398 640 webserver.cc:460] Webserver started at http://127.0.98.2:35301/ using document root <none> and password file <none>
I20260926 20:55:52.448689 640 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:52.448761 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:52.448921 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:52.450024 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "6f6949fb64544b13aaf7a09643e2bf90"
format_stamp: "Formatted at 2026-09-26 20:55:52 on dist-test-slave-fl7g"
server_key: "8457d72d3e8933350e7dd0e3d586df56"
server_key_iv: "2b3a9574fd16380d0eed010e099c9122"
server_key_version: "encryptionkey@0"
I20260926 20:55:52.450412 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6f6949fb64544b13aaf7a09643e2bf90"
format_stamp: "Formatted at 2026-09-26 20:55:52 on dist-test-slave-fl7g"
server_key: "8457d72d3e8933350e7dd0e3d586df56"
server_key_iv: "2b3a9574fd16380d0eed010e099c9122"
server_key_version: "encryptionkey@0"
I20260926 20:55:52.451833 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:55:52.452797 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:52.453029 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:55:52.453155 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "6f6949fb64544b13aaf7a09643e2bf90"
format_stamp: "Formatted at 2026-09-26 20:55:52 on dist-test-slave-fl7g"
server_key: "8457d72d3e8933350e7dd0e3d586df56"
server_key_iv: "2b3a9574fd16380d0eed010e099c9122"
server_key_version: "encryptionkey@0"
I20260926 20:55:52.453265 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:52.470866 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:52.473377 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:52.473567 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:52.473857 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:52.474273 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:52.474334 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:52.474377 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:52.474404 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:52.480943 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35131
I20260926 20:55:52.480989 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35131 every 8 connection(s)
I20260926 20:55:52.481366 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:55:52.484359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 640
I20260926 20:55:52.484519 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:55:52.484695 392 external_mini_cluster.cc:1428] Setting key ae7dfd0714a3191f2457fac9ffacf57c
Sep 26 20:55:52 dist-test-slave-fl7g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:55:52.492568 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:52.482279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55287 (local address 127.0.98.62:37011)
0926 20:55:52.482340 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:52.482349 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:52.482565 (+ 216us) server_negotiation.cc:407] Connection header received
0926 20:55:52.483066 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:52.483068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:52.483079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:52.483146 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:52.484115 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:52.484733 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:52.485953 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:52.486280 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:52.486349 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:52.489337 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:52.489339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:52.489339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:52.489354 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:52.490984 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:52.491477 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:52.491478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:52.491479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:52.491547 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:52.491826 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:52.491827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:52.491827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:52.491954 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:52.492033 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:52.492339 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 20:55:52.492462 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 20:55:52.492787 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:52.482232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37011 (local address 127.0.98.2:55287)
0926 20:55:52.482400 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:52.482444 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:52.482949 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:52.483228 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:52.483232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:52.483824 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:52.484026 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:52.484030 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:52.484861 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:52.484863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:52.485863 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:52.485866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:52.486371 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:52.486372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:52.486492 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:52.487065 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:52.487074 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:52.489193 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:52.491128 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:52.491137 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:52.491140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:52.491390 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:52.491628 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:52.491629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:52.491629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:52.491753 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:52.492023 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:52.492038 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:52.492189 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 20:55:52.492304 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:55:52.492526 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260926 20:55:52.493270 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37011
I20260926 20:55:52.493369 769 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 20:55:52.493597 769 heartbeater.cc:511] Master 127.0.98.62:37011 requested a full tablet report, sending...
I20260926 20:55:52.494091 444 ts_manager.cc:194] Registered new tserver with Master: 6f6949fb64544b13aaf7a09643e2bf90 (127.0.98.2:35131)
I20260926 20:55:52.494506 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:55:52.511696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:52.596131 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:52.596383 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:52.596488 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:52.597764 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:52.597826 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:52.597855 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:55:52.599373 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:52.599606 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:52.599802 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:52.605165 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:52.605173 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:52.605201 774 server_base.cc:1034] running on GCE node
W20260926 20:55:52.605170 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:52.605661 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:52.605921 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:52.607079 774 hybrid_clock.cc:648] HybridClock initialized: now 1790456152607064 us; error 29 us; skew 500 ppm
Sep 26 20:55:52 dist-test-slave-fl7g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:52.609666 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:55:52.610210 774 webserver.cc:460] Webserver started at http://127.0.98.3:45407/ using document root <none> and password file <none>
I20260926 20:55:52.610455 774 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:52.610532 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:52.610682 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:52.611761 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "246f7795e7ad4eb099f9624881326c77"
format_stamp: "Formatted at 2026-09-26 20:55:52 on dist-test-slave-fl7g"
server_key: "20c79e908cda7b2a07f275a4038fe6cb"
server_key_iv: "0f898deac7da0cd29696a0cc3d0ab1e6"
server_key_version: "encryptionkey@0"
I20260926 20:55:52.612142 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "246f7795e7ad4eb099f9624881326c77"
format_stamp: "Formatted at 2026-09-26 20:55:52 on dist-test-slave-fl7g"
server_key: "20c79e908cda7b2a07f275a4038fe6cb"
server_key_iv: "0f898deac7da0cd29696a0cc3d0ab1e6"
server_key_version: "encryptionkey@0"
I20260926 20:55:52.613634 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:55:52.614372 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:52.614595 774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:55:52.614699 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "246f7795e7ad4eb099f9624881326c77"
format_stamp: "Formatted at 2026-09-26 20:55:52 on dist-test-slave-fl7g"
server_key: "20c79e908cda7b2a07f275a4038fe6cb"
server_key_iv: "0f898deac7da0cd29696a0cc3d0ab1e6"
server_key_version: "encryptionkey@0"
I20260926 20:55:52.614789 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:52.635099 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:52.637997 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:52.638191 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:52.638453 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:52.638796 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:52.638856 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:52.638902 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:52.638926 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:52.645726 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46151
I20260926 20:55:52.645771 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46151 every 8 connection(s)
I20260926 20:55:52.646167 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:55:52.647414 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 774
I20260926 20:55:52.647495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:55:52.647763 392 external_mini_cluster.cc:1428] Setting key 0aedb4baa6f051002dd85f8e29a5cce1
Sep 26 20:55:52 dist-test-slave-fl7g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:55:52.656011 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:52.646973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46147 (local address 127.0.98.62:37011)
0926 20:55:52.647035 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:52.647037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:52.647271 (+ 234us) server_negotiation.cc:407] Connection header received
0926 20:55:52.647931 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:52.647933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:52.647946 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:52.648005 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:52.649124 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:52.649632 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:52.650662 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:52.650879 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:52.650973 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:52.653156 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:52.653157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:52.653158 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:52.653166 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:52.654518 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:52.655006 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:52.655007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:52.655007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:52.655073 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:52.655351 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:52.655352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:52.655352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:52.655470 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:52.655548 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:52.655835 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 20:55:52.655917 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:55:52.656286 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:52.646957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37011 (local address 127.0.98.3:46147)
0926 20:55:52.647138 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:52.647183 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:52.647762 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:52.648096 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:52.648099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:52.648821 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:52.649033 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:52.649038 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:52.649751 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:52.649751 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:52.650570 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:52.650572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:52.650982 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:52.650983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:52.651111 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:52.651592 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:52.651600 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:52.653055 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:52.654680 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:52.654688 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:52.654690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:52.654926 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:52.655151 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:52.655152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:52.655152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:52.655276 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:52.655543 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:52.655550 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:52.655698 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 20:55:52.655805 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 20:55:52.655973 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260926 20:55:52.656802 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37011
I20260926 20:55:52.656901 903 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:52.657043 903 heartbeater.cc:511] Master 127.0.98.62:37011 requested a full tablet report, sending...
I20260926 20:55:52.657440 444 ts_manager.cc:194] Registered new tserver with Master: 246f7795e7ad4eb099f9624881326c77 (127.0.98.3:46151)
I20260926 20:55:52.657804 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46147
I20260926 20:55:52.659724 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:55:52 dist-test-slave-fl7g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456152, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:55:52 dist-test-slave-fl7g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456152, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:55:52.683250 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:52.675384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48124 (local address 127.0.98.1:38963)
0926 20:55:52.675568 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:52.675597 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:52.675610 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:55:52.675686 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:52.675689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:52.675708 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:52.675786 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:52.676372 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:52.677167 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:52.677941 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:52.678262 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:52.678356 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:52.680636 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:52.680644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:52.680644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:52.680652 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:52.682019 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:52.682394 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:52.682398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:52.682398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:52.682461 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:52.682664 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:52.682665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:52.682666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:52.682810 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:52.682986 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:52.683032 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 20:55:52.683115 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":75,"threads_started":1}
W20260926 20:55:52.683612 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48124
Sep 26 20:55:52 dist-test-slave-fl7g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:55:52 dist-test-slave-fl7g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:55:52.706059 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:52.698626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48136 (local address 127.0.98.1:38963)
0926 20:55:52.698694 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:52.698696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:52.698821 (+ 125us) server_negotiation.cc:407] Connection header received
0926 20:55:52.698853 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:52.698854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:52.698867 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:52.698938 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:52.699562 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:52.700074 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:52.700975 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:52.701185 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:52.701253 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:52.703622 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:52.703624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:52.703624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:52.703633 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:52.705030 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:52.705389 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:52.705390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:52.705391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:52.705441 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:52.705658 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:52.705659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:52.705659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:52.705752 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:52.705843 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:52.705928 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 20:55:52.705988 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:55:52.706734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 506
I20260926 20:55:52.711536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 640
I20260926 20:55:52.717664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 774
I20260926 20:55:52.722198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 412
2026-09-26T20:55:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1930 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:55:52 dist-test-slave-fl7g krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:55:52 dist-test-slave-fl7g krb5kdc[917](info): set up 2 sockets
Sep 26 20:55:52 dist-test-slave-fl7g krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260926 20:55:53.776023 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:55:53 dist-test-slave-fl7g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:55:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:55:53.902931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45087
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:53.984287 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:53.984561 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:53.984604 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:53.985792 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:53.985873 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:53.985898 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:53.985927 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:53.985958 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:53.987457 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:53.987742 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:53.987985 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:53.993136 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:53.993145 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:53.993254 930 server_base.cc:1034] running on GCE node
W20260926 20:55:53.993155 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:53.993595 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:53.993948 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:53.995124 930 hybrid_clock.cc:648] HybridClock initialized: now 1790456153995094 us; error 42 us; skew 500 ppm
Sep 26 20:55:53 dist-test-slave-fl7g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:53.997748 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:55:53.998270 930 webserver.cc:460] Webserver started at http://127.0.98.62:36359/ using document root <none> and password file <none>
I20260926 20:55:53.998528 930 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:53.998598 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:53.998755 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:53.999841 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "82df3e5c44fa4d40b75975baa95b2f30"
format_stamp: "Formatted at 2026-09-26 20:55:53 on dist-test-slave-fl7g"
server_key: "edac04d222af8d9d1999bc7201152675"
server_key_iv: "a96e011dbcd56af728d52e833f402ccd"
server_key_version: "encryptionkey@0"
I20260926 20:55:54.000213 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "82df3e5c44fa4d40b75975baa95b2f30"
format_stamp: "Formatted at 2026-09-26 20:55:53 on dist-test-slave-fl7g"
server_key: "edac04d222af8d9d1999bc7201152675"
server_key_iv: "a96e011dbcd56af728d52e833f402ccd"
server_key_version: "encryptionkey@0"
I20260926 20:55:54.001627 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:55:54.002560 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:54.002794 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:55:54.002907 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "82df3e5c44fa4d40b75975baa95b2f30"
format_stamp: "Formatted at 2026-09-26 20:55:53 on dist-test-slave-fl7g"
server_key: "edac04d222af8d9d1999bc7201152675"
server_key_iv: "a96e011dbcd56af728d52e833f402ccd"
server_key_version: "encryptionkey@0"
I20260926 20:55:54.003003 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:54.017261 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:54.019508 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:54.019702 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:54.023574 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45087
I20260926 20:55:54.023648 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45087 every 8 connection(s)
I20260926 20:55:54.024035 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:55:54.025069 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:54.027400 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30: Bootstrap starting.
I20260926 20:55:54.027848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 930
I20260926 20:55:54.027978 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:55:54.028040 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:54.028180 392 external_mini_cluster.cc:1428] Setting key c7862ef80885a7b733b396582b3f0c5f
I20260926 20:55:54.028398 998 log.cc:824] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:54.029315 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30: No bootstrap required, opened a new log
I20260926 20:55:54.031693 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82df3e5c44fa4d40b75975baa95b2f30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45087 } }
I20260926 20:55:54.031879 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:54.031914 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82df3e5c44fa4d40b75975baa95b2f30, State: Initialized, Role: FOLLOWER
I20260926 20:55:54.032068 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82df3e5c44fa4d40b75975baa95b2f30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45087 } }
I20260926 20:55:54.032147 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:55:54.032182 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:55:54.032224 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:54.032913 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82df3e5c44fa4d40b75975baa95b2f30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45087 } }
I20260926 20:55:54.033063 998 leader_election.cc:302] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82df3e5c44fa4d40b75975baa95b2f30; no voters:
I20260926 20:55:54.033277 998 leader_election.cc:288] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:55:54.033317 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:54.033484 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30 [term 1 LEADER]: Becoming Leader. State: Replica: 82df3e5c44fa4d40b75975baa95b2f30, State: Running, Role: LEADER
I20260926 20:55:54.033569 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82df3e5c44fa4d40b75975baa95b2f30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45087 } }
I20260926 20:55:54.033599 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:54.033912 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82df3e5c44fa4d40b75975baa95b2f30. Latest consensus state: current_term: 1 leader_uuid: "82df3e5c44fa4d40b75975baa95b2f30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82df3e5c44fa4d40b75975baa95b2f30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45087 } } }
I20260926 20:55:54.033991 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30 [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:54.035408 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:55:54.035398 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82df3e5c44fa4d40b75975baa95b2f30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82df3e5c44fa4d40b75975baa95b2f30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45087 } } }
I20260926 20:55:54.035502 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30 [sys.catalog]: This master's current role is: LEADER
Sep 26 20:55:54 dist-test-slave-fl7g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:55:54.035974 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:55:54.038235 1008 catalog_manager.cc:1347] Generated new cluster ID: 4dc51d299f5547afa1c2c92020090df0
I20260926 20:55:54.038280 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:55:54.040254 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.028784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44870 (local address 127.0.98.62:45087)
0926 20:55:54.028944 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:54.028955 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:54.028999 (+ 44us) server_negotiation.cc:407] Connection header received
0926 20:55:54.029571 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:54.029578 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:54.029632 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:54.029754 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:54.030530 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.031421 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.032520 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.032945 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.033084 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.036282 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:54.036287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:54.036287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:54.036302 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:54.038628 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:54.039050 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:54.039053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:54.039054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:54.039153 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:54.039367 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:54.039368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:54.039368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:54.039647 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:54.039795 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:54.039918 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 20:55:54.040049 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260926 20:55:54.062371 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:55:54.062976 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:55:54.066407 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82df3e5c44fa4d40b75975baa95b2f30: Generated new TSK 0
I20260926 20:55:54.066644 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:55:54.095551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45087
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:54.180327 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:54.180612 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:54.180641 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:54.181969 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:54.182027 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:54.182060 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:55:54.183702 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45087
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:54.184013 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:54.184289 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:55:54.189831 1024 server_base.cc:1034] running on GCE node
W20260926 20:55:54.189777 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:54.189874 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:54.189786 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:54.190184 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:54.190454 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:54.191601 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790456154191591 us; error 32 us; skew 500 ppm
Sep 26 20:55:54 dist-test-slave-fl7g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:54.198027 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:55:54.198565 1024 webserver.cc:460] Webserver started at http://127.0.98.1:40255/ using document root <none> and password file <none>
I20260926 20:55:54.198774 1024 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:54.198817 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:54.198920 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:54.200042 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "e08a418e4c2041a4a79e132d3b2918a9"
format_stamp: "Formatted at 2026-09-26 20:55:54 on dist-test-slave-fl7g"
server_key: "72f0f29d5a094cd9fa3195246002e40c"
server_key_iv: "974d39223d5ce7dbd732787e2f166287"
server_key_version: "encryptionkey@0"
I20260926 20:55:54.200380 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e08a418e4c2041a4a79e132d3b2918a9"
format_stamp: "Formatted at 2026-09-26 20:55:54 on dist-test-slave-fl7g"
server_key: "72f0f29d5a094cd9fa3195246002e40c"
server_key_iv: "974d39223d5ce7dbd732787e2f166287"
server_key_version: "encryptionkey@0"
I20260926 20:55:54.201874 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:55:54.202646 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:54.202826 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:55:54.202936 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "e08a418e4c2041a4a79e132d3b2918a9"
format_stamp: "Formatted at 2026-09-26 20:55:54 on dist-test-slave-fl7g"
server_key: "72f0f29d5a094cd9fa3195246002e40c"
server_key_iv: "974d39223d5ce7dbd732787e2f166287"
server_key_version: "encryptionkey@0"
I20260926 20:55:54.203028 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:54.219587 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:54.221632 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:54.221808 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:54.222064 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:54.222411 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:54.222474 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:54.222517 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:54.222543 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:54.228658 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37949
I20260926 20:55:54.228704 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37949 every 8 connection(s)
I20260926 20:55:54.229041 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:55:54.230921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 1024
I20260926 20:55:54.230998 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:55:54.231175 392 external_mini_cluster.cc:1428] Setting key 58dad8b7702366f3d01bbf0e4a28ce26
Sep 26 20:55:54 dist-test-slave-fl7g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 20:55:54.240592 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.229813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33743 (local address 127.0.98.62:45087)
0926 20:55:54.229877 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:54.229879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:54.230129 (+ 250us) server_negotiation.cc:407] Connection header received
0926 20:55:54.230633 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:54.230634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:54.230645 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:54.230697 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:54.231922 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.232714 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.234386 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.234853 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.234923 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.237476 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:54.237478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:54.237479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:54.237490 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:54.239123 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:54.239531 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:54.239532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:54.239533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:54.239600 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:54.239864 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:54.239865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:54.239866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:54.240018 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:54.240092 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:54.240342 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 20:55:54.240469 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:55:54.240746 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.229791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45087 (local address 127.0.98.1:33743)
0926 20:55:54.229968 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:54.230034 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:54.230539 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:54.230773 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:54.230777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:54.231575 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:54.231805 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:54.231811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.232806 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.232807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:54.234274 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:54.234277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.234951 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.234952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:54.235073 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.235680 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:54.235689 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:54.237375 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:54.239209 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:54.239216 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:54.239218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:54.239446 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:54.239674 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:54.239674 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:54.239674 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:54.239793 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:54.240085 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:54.240089 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:54.240211 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 20:55:54.240296 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 20:55:54.240502 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
I20260926 20:55:54.241245 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45087
I20260926 20:55:54.241362 1153 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:54.241550 1153 heartbeater.cc:511] Master 127.0.98.62:45087 requested a full tablet report, sending...
I20260926 20:55:54.242127 962 ts_manager.cc:194] Registered new tserver with Master: e08a418e4c2041a4a79e132d3b2918a9 (127.0.98.1:37949)
I20260926 20:55:54.242766 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:55:54.260396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45087
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:54.343310 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:54.343590 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:54.343632 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:54.344934 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:54.345002 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:54.345049 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:55:54.346818 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45087
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:54.347110 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:54.347370 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:54.352725 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:54.352737 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:54.352732 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:54.352869 1158 server_base.cc:1034] running on GCE node
I20260926 20:55:54.353237 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:54.353541 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:54.354730 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790456154354724 us; error 50 us; skew 500 ppm
Sep 26 20:55:54 dist-test-slave-fl7g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:54.357597 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:55:54.358162 1158 webserver.cc:460] Webserver started at http://127.0.98.2:33409/ using document root <none> and password file <none>
I20260926 20:55:54.358425 1158 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:54.358498 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:54.358649 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:54.359750 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "887ce47dc98b45daa48a81e2d854df24"
format_stamp: "Formatted at 2026-09-26 20:55:54 on dist-test-slave-fl7g"
server_key: "8eb712bf661a318e5e3317cf31e33280"
server_key_iv: "1f5893e308e8828a2ce2fa54ccd5e1b3"
server_key_version: "encryptionkey@0"
I20260926 20:55:54.360184 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "887ce47dc98b45daa48a81e2d854df24"
format_stamp: "Formatted at 2026-09-26 20:55:54 on dist-test-slave-fl7g"
server_key: "8eb712bf661a318e5e3317cf31e33280"
server_key_iv: "1f5893e308e8828a2ce2fa54ccd5e1b3"
server_key_version: "encryptionkey@0"
I20260926 20:55:54.361691 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:55:54.362600 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:54.362823 1158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:55:54.362926 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "887ce47dc98b45daa48a81e2d854df24"
format_stamp: "Formatted at 2026-09-26 20:55:54 on dist-test-slave-fl7g"
server_key: "8eb712bf661a318e5e3317cf31e33280"
server_key_iv: "1f5893e308e8828a2ce2fa54ccd5e1b3"
server_key_version: "encryptionkey@0"
I20260926 20:55:54.363013 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:54.387795 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:54.390318 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:54.390502 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:54.390772 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:54.391103 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:54.391162 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:54.391206 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:54.391230 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:54.397193 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40333
I20260926 20:55:54.397235 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40333 every 8 connection(s)
I20260926 20:55:54.397632 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:55:54 dist-test-slave-fl7g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:55:54.405792 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 1158
I20260926 20:55:54.405917 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:55:54.406127 392 external_mini_cluster.cc:1428] Setting key a49d38954c301ba474193de51bc918aa
I20260926 20:55:54.407991 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.398504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53299 (local address 127.0.98.62:45087)
0926 20:55:54.398593 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:54.398596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:54.398845 (+ 249us) server_negotiation.cc:407] Connection header received
0926 20:55:54.399340 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:54.399341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:54.399352 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:54.399413 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:54.400454 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.400992 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.402003 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.402206 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.402267 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.404637 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:54.404639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:54.404639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:54.404646 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:54.406355 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:54.406915 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:54.406917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:54.406917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:54.407005 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:54.407284 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:54.407285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:54.407285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:54.407411 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:54.407508 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:54.407805 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 20:55:54.407891 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:55:54.408358 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.398461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45087 (local address 127.0.98.2:53299)
0926 20:55:54.398649 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:54.398688 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:54.399249 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:54.399486 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:54.399490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:54.400085 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:54.400309 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:54.400314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.401084 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.401084 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:54.401921 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:54.401923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.402316 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.402317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:54.402449 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.402997 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:54.403006 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:54.404529 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:54.406468 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:54.406474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:54.406476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:54.406824 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:54.407061 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:54.407062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:54.407062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:54.407218 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:54.407510 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:54.407517 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:54.407666 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 20:55:54.407771 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 20:55:54.407932 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":106,"threads_started":1}
I20260926 20:55:54.409060 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45087
I20260926 20:55:54.409189 1287 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:54.409382 1287 heartbeater.cc:511] Master 127.0.98.62:45087 requested a full tablet report, sending...
I20260926 20:55:54.409786 962 ts_manager.cc:194] Registered new tserver with Master: 887ce47dc98b45daa48a81e2d854df24 (127.0.98.2:40333)
I20260926 20:55:54.410182 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53299
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:55:54.435207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45087
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:54.517463 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:54.517729 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:54.517774 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:54.519033 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:54.519099 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:54.519150 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:55:54.520718 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45087
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:54.520990 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:54.521219 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:55:54.526664 1292 server_base.cc:1034] running on GCE node
W20260926 20:55:54.526657 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:54.526665 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:54.526824 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:54.527015 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:54.527225 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:54.528440 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790456154528427 us; error 39 us; skew 500 ppm
Sep 26 20:55:54 dist-test-slave-fl7g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:54.531073 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:55:54.531679 1292 webserver.cc:460] Webserver started at http://127.0.98.3:43393/ using document root <none> and password file <none>
I20260926 20:55:54.531939 1292 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:54.532009 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:54.532164 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:54.533427 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "51a0d14dd9dc423cb658ecb8aaa2ed70"
format_stamp: "Formatted at 2026-09-26 20:55:54 on dist-test-slave-fl7g"
server_key: "8c709da6c5c2e21e6a8bc3c3fe910882"
server_key_iv: "b1e5f2c8a54974605b024bd2868fa68d"
server_key_version: "encryptionkey@0"
I20260926 20:55:54.533807 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "51a0d14dd9dc423cb658ecb8aaa2ed70"
format_stamp: "Formatted at 2026-09-26 20:55:54 on dist-test-slave-fl7g"
server_key: "8c709da6c5c2e21e6a8bc3c3fe910882"
server_key_iv: "b1e5f2c8a54974605b024bd2868fa68d"
server_key_version: "encryptionkey@0"
I20260926 20:55:54.535290 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:55:54.536083 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:54.536280 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:55:54.536386 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "51a0d14dd9dc423cb658ecb8aaa2ed70"
format_stamp: "Formatted at 2026-09-26 20:55:54 on dist-test-slave-fl7g"
server_key: "8c709da6c5c2e21e6a8bc3c3fe910882"
server_key_iv: "b1e5f2c8a54974605b024bd2868fa68d"
server_key_version: "encryptionkey@0"
I20260926 20:55:54.536513 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:54.560475 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:54.562805 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:54.563090 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:54.563374 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:54.563712 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:54.563771 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:54.563817 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:54.563848 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:54.571086 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40399
I20260926 20:55:54.571174 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40399 every 8 connection(s)
I20260926 20:55:54.571527 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:55:54 dist-test-slave-fl7g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:55:54.580780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 1292
I20260926 20:55:54.580907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:55:54.581085 392 external_mini_cluster.cc:1428] Setting key a65ab78cefe8c83440a1e9e9d4bb22a8
I20260926 20:55:54.581945 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.572442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59547 (local address 127.0.98.62:45087)
0926 20:55:54.572516 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:54.572518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:54.572725 (+ 207us) server_negotiation.cc:407] Connection header received
0926 20:55:54.573276 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:54.573277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:54.573289 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:54.573347 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:54.574350 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.574860 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.575929 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.576154 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.576256 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.578715 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:54.578718 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:54.578718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:54.578725 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:54.580047 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:54.580696 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:54.580697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:54.580698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:54.580771 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:54.581086 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:54.581087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:54.581087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:54.581266 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:54.581328 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:54.581719 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 20:55:54.581822 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:55:54.582124 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.572366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45087 (local address 127.0.98.3:59547)
0926 20:55:54.572594 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:54.572633 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:54.573168 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:54.573432 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:54.573436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:54.574041 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:54.574243 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:54.574248 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.574975 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.574976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:54.575816 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:54.575818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.576267 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.576268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:54.576455 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.576979 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:54.576988 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:54.578605 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:54.580190 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:54.580200 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:54.580203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:54.580569 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:54.580851 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:54.580852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:54.580852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:54.581021 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:54.581336 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:54.581343 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:54.581541 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 20:55:54.581656 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:55:54.581871 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":149,"threads_started":1}
I20260926 20:55:54.582670 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45087
I20260926 20:55:54.582756 1421 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:54.582907 1421 heartbeater.cc:511] Master 127.0.98.62:45087 requested a full tablet report, sending...
I20260926 20:55:54.583244 962 ts_manager.cc:194] Registered new tserver with Master: 51a0d14dd9dc423cb658ecb8aaa2ed70 (127.0.98.3:40399)
I20260926 20:55:54.583585 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59547
I20260926 20:55:54.593106 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:55:54.601415 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.594888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44886 (local address 127.0.98.62:45087)
0926 20:55:54.594976 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:54.594979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:54.595041 (+ 62us) server_negotiation.cc:407] Connection header received
0926 20:55:54.595126 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:54.595128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:54.595138 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:54.595213 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:54.595807 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.596320 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.597128 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.597332 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.597421 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.598679 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:54.598682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:54.598682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:54.598690 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:54.600044 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:54.600517 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:54.600518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:54.600523 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:54.600581 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:54.600793 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:54.600794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:54.600795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:54.600975 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:54.601100 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:54.601213 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 20:55:54.601307 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:55:54.604242 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:55:54.606249 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:55:54.612893 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.608864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37949 (local address 127.0.0.1:46644)
0926 20:55:54.608985 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:54.609028 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:54.609102 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:54.609505 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:54.609508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:54.609510 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:54.609625 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:54.609628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.610714 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.610715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:54.611815 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:54.611817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.612579 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.612579 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:54.612655 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.612670 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 20:55:54.612747 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:55:54.612812 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":70,"threads_started":1}
I20260926 20:55:54.612890 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.609140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46644 (local address 127.0.98.1:37949)
0926 20:55:54.609313 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:54.609318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:54.609331 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:55:54.609371 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:54.609373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:54.609387 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:54.609500 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:54.609716 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.610608 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.611886 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.612512 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.612581 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.612644 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:54.612762 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 20:55:54.612803 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20260926 20:55:54.613473 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.608869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40333 (local address 127.0.0.1:50020)
0926 20:55:54.609619 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:54.609629 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:54.609676 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:54.609970 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:54.609971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:54.609973 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:54.610012 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:54.610013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.611270 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.611271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:54.612410 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:54.612413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.613250 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.613251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:54.613310 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.613315 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:55:54.613375 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 20:55:54.613412 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":93,"threads_started":1}
I20260926 20:55:54.613523 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.609052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50020 (local address 127.0.98.2:40333)
0926 20:55:54.609332 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:54.609336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:54.609735 (+ 399us) server_negotiation.cc:407] Connection header received
0926 20:55:54.609803 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:54.609805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:54.609821 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:54.609894 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:54.610109 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.611162 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.612515 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.613172 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.613231 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.613262 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:54.613406 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 20:55:54.613448 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":91,"threads_started":1}
I20260926 20:55:54.613903 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.608864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40399 (local address 127.0.0.1:41958)
0926 20:55:54.609231 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:54.609241 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:54.609286 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:54.609511 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:54.609512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:54.609515 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:54.609593 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:54.609595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.610891 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.610892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:54.611902 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:54.611904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.612919 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.612920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:54.613719 (+ 799us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.613724 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:55:54.613784 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 20:55:54.613814 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":3,"thread_start_us":88,"threads_started":1}
I20260926 20:55:54.613968 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.609117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41958 (local address 127.0.98.3:40399)
0926 20:55:54.609295 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:54.609300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:54.609311 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:55:54.609350 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:54.609352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:54.609367 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:54.609503 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:54.609677 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.610779 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.612000 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.612776 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.612836 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.612886 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:54.613845 (+ 959us) server_negotiation.cc:299] Negotiation successful
0926 20:55:54.613892 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260926 20:55:54.613826 1087 tablet_service.cc:1467] Processing CreateTablet for tablet b17b3628b2484ce29ec9926d564bc33c (DEFAULT_TABLE table=test-table [id=a4a0b24d25c64644a00d432ff817fb88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:54.614245 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b17b3628b2484ce29ec9926d564bc33c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:54.614832 1346 tablet_service.cc:1467] Processing CreateTablet for tablet b17b3628b2484ce29ec9926d564bc33c (DEFAULT_TABLE table=test-table [id=a4a0b24d25c64644a00d432ff817fb88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:54.615150 1346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b17b3628b2484ce29ec9926d564bc33c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:54.617344 1221 tablet_service.cc:1467] Processing CreateTablet for tablet b17b3628b2484ce29ec9926d564bc33c (DEFAULT_TABLE table=test-table [id=a4a0b24d25c64644a00d432ff817fb88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:54.617694 1438 tablet_bootstrap.cc:492] T b17b3628b2484ce29ec9926d564bc33c P e08a418e4c2041a4a79e132d3b2918a9: Bootstrap starting.
I20260926 20:55:54.617724 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b17b3628b2484ce29ec9926d564bc33c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:54.618319 1439 tablet_bootstrap.cc:492] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70: Bootstrap starting.
I20260926 20:55:54.618381 1438 tablet_bootstrap.cc:654] T b17b3628b2484ce29ec9926d564bc33c P e08a418e4c2041a4a79e132d3b2918a9: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:54.619030 1439 tablet_bootstrap.cc:654] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:54.619020 1438 log.cc:824] T b17b3628b2484ce29ec9926d564bc33c P e08a418e4c2041a4a79e132d3b2918a9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:54.619547 1439 log.cc:824] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:54.619913 1438 tablet_bootstrap.cc:492] T b17b3628b2484ce29ec9926d564bc33c P e08a418e4c2041a4a79e132d3b2918a9: No bootstrap required, opened a new log
I20260926 20:55:54.620013 1438 ts_tablet_manager.cc:1397] T b17b3628b2484ce29ec9926d564bc33c P e08a418e4c2041a4a79e132d3b2918a9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:55:54.620297 1439 tablet_bootstrap.cc:492] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70: No bootstrap required, opened a new log
I20260926 20:55:54.620373 1439 ts_tablet_manager.cc:1397] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:55:54.621086 1442 tablet_bootstrap.cc:492] T b17b3628b2484ce29ec9926d564bc33c P 887ce47dc98b45daa48a81e2d854df24: Bootstrap starting.
I20260926 20:55:54.621757 1442 tablet_bootstrap.cc:654] T b17b3628b2484ce29ec9926d564bc33c P 887ce47dc98b45daa48a81e2d854df24: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:54.621831 1438 raft_consensus.cc:348] T b17b3628b2484ce29ec9926d564bc33c P e08a418e4c2041a4a79e132d3b2918a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a418e4c2041a4a79e132d3b2918a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37949 } } peers { permanent_uuid: "887ce47dc98b45daa48a81e2d854df24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40333 } } peers { permanent_uuid: "51a0d14dd9dc423cb658ecb8aaa2ed70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } }
I20260926 20:55:54.622031 1438 raft_consensus.cc:374] T b17b3628b2484ce29ec9926d564bc33c P e08a418e4c2041a4a79e132d3b2918a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:54.622076 1438 raft_consensus.cc:729] T b17b3628b2484ce29ec9926d564bc33c P e08a418e4c2041a4a79e132d3b2918a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08a418e4c2041a4a79e132d3b2918a9, State: Initialized, Role: FOLLOWER
I20260926 20:55:54.622189 1438 consensus_queue.cc:260] T b17b3628b2484ce29ec9926d564bc33c P e08a418e4c2041a4a79e132d3b2918a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a418e4c2041a4a79e132d3b2918a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37949 } } peers { permanent_uuid: "887ce47dc98b45daa48a81e2d854df24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40333 } } peers { permanent_uuid: "51a0d14dd9dc423cb658ecb8aaa2ed70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } }
I20260926 20:55:54.622466 1438 ts_tablet_manager.cc:1428] T b17b3628b2484ce29ec9926d564bc33c P e08a418e4c2041a4a79e132d3b2918a9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:55:54.622599 1153 heartbeater.cc:503] Master 127.0.98.62:45087 was elected leader, sending a full tablet report...
I20260926 20:55:54.622710 1442 log.cc:824] T b17b3628b2484ce29ec9926d564bc33c P 887ce47dc98b45daa48a81e2d854df24: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:54.623131 1439 raft_consensus.cc:348] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a418e4c2041a4a79e132d3b2918a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37949 } } peers { permanent_uuid: "887ce47dc98b45daa48a81e2d854df24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40333 } } peers { permanent_uuid: "51a0d14dd9dc423cb658ecb8aaa2ed70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } }
I20260926 20:55:54.623337 1439 raft_consensus.cc:374] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:54.623384 1439 raft_consensus.cc:729] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a0d14dd9dc423cb658ecb8aaa2ed70, State: Initialized, Role: FOLLOWER
I20260926 20:55:54.623494 1439 consensus_queue.cc:260] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a418e4c2041a4a79e132d3b2918a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37949 } } peers { permanent_uuid: "887ce47dc98b45daa48a81e2d854df24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40333 } } peers { permanent_uuid: "51a0d14dd9dc423cb658ecb8aaa2ed70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } }
I20260926 20:55:54.623591 1442 tablet_bootstrap.cc:492] T b17b3628b2484ce29ec9926d564bc33c P 887ce47dc98b45daa48a81e2d854df24: No bootstrap required, opened a new log
I20260926 20:55:54.623680 1442 ts_tablet_manager.cc:1397] T b17b3628b2484ce29ec9926d564bc33c P 887ce47dc98b45daa48a81e2d854df24: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:55:54.623768 1439 ts_tablet_manager.cc:1428] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 20:55:54.623919 1421 heartbeater.cc:503] Master 127.0.98.62:45087 was elected leader, sending a full tablet report...
I20260926 20:55:54.625550 1442 raft_consensus.cc:348] T b17b3628b2484ce29ec9926d564bc33c P 887ce47dc98b45daa48a81e2d854df24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a418e4c2041a4a79e132d3b2918a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37949 } } peers { permanent_uuid: "887ce47dc98b45daa48a81e2d854df24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40333 } } peers { permanent_uuid: "51a0d14dd9dc423cb658ecb8aaa2ed70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } }
I20260926 20:55:54.625689 1442 raft_consensus.cc:374] T b17b3628b2484ce29ec9926d564bc33c P 887ce47dc98b45daa48a81e2d854df24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:54.625710 1442 raft_consensus.cc:729] T b17b3628b2484ce29ec9926d564bc33c P 887ce47dc98b45daa48a81e2d854df24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 887ce47dc98b45daa48a81e2d854df24, State: Initialized, Role: FOLLOWER
I20260926 20:55:54.625790 1442 consensus_queue.cc:260] T b17b3628b2484ce29ec9926d564bc33c P 887ce47dc98b45daa48a81e2d854df24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a418e4c2041a4a79e132d3b2918a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37949 } } peers { permanent_uuid: "887ce47dc98b45daa48a81e2d854df24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40333 } } peers { permanent_uuid: "51a0d14dd9dc423cb658ecb8aaa2ed70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } }
I20260926 20:55:54.625990 1442 ts_tablet_manager.cc:1428] T b17b3628b2484ce29ec9926d564bc33c P 887ce47dc98b45daa48a81e2d854df24: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:55:54.626040 1287 heartbeater.cc:503] Master 127.0.98.62:45087 was elected leader, sending a full tablet report...
W20260926 20:55:54.648254 1288 tablet.cc:2369] T b17b3628b2484ce29ec9926d564bc33c P 887ce47dc98b45daa48a81e2d854df24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:55:54.677191 1445 raft_consensus.cc:482] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:55:54.677322 1445 raft_consensus.cc:504] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a418e4c2041a4a79e132d3b2918a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37949 } } peers { permanent_uuid: "887ce47dc98b45daa48a81e2d854df24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40333 } } peers { permanent_uuid: "51a0d14dd9dc423cb658ecb8aaa2ed70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } }
I20260926 20:55:54.677762 1445 leader_election.cc:288] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e08a418e4c2041a4a79e132d3b2918a9 (127.0.98.1:37949), 887ce47dc98b45daa48a81e2d854df24 (127.0.98.2:40333)
I20260926 20:55:54.681005 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.678152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36177 (local address 127.0.98.2:40333)
0926 20:55:54.678206 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:54.678208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:54.678215 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:55:54.678249 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:54.678250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:54.678262 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:54.678339 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:54.678565 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.679060 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.680028 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.680608 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.680698 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.680709 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:54.680883 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 20:55:54.680921 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:55:54.681022 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.678126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55823 (local address 127.0.98.1:37949)
0926 20:55:54.678180 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:54.678182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:54.678329 (+ 147us) server_negotiation.cc:407] Connection header received
0926 20:55:54.678420 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:54.678421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:54.678433 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:54.678535 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:54.678703 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.679160 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.680108 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.680665 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.680745 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.680756 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:54.680908 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 20:55:54.680947 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 20:55:54.681047 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.678029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40333 (local address 127.0.98.3:36177)
0926 20:55:54.678092 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:54.678104 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:54.678147 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:54.678365 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:54.678366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:54.678368 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:54.678482 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:54.678484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.679159 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.679160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:54.679926 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:54.679928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.680699 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.680699 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:54.680787 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.680793 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:55:54.680863 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 20:55:54.680910 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 20:55:54.681259 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.678059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37949 (local address 127.0.98.3:55823)
0926 20:55:54.678237 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:54.678247 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:54.678338 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:54.678544 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:54.678545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:54.678547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:54.678627 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:54.678629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.679253 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.679254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:54.680011 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:54.680013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.680758 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.680759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:54.680833 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.680836 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:55:54.680903 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 20:55:54.680944 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":131,"threads_started":1}
I20260926 20:55:54.681452 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17b3628b2484ce29ec9926d564bc33c" candidate_uuid: "51a0d14dd9dc423cb658ecb8aaa2ed70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "887ce47dc98b45daa48a81e2d854df24" is_pre_election: true
I20260926 20:55:54.681650 1241 raft_consensus.cc:2454] T b17b3628b2484ce29ec9926d564bc33c P 887ce47dc98b45daa48a81e2d854df24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51a0d14dd9dc423cb658ecb8aaa2ed70 in term 0.
I20260926 20:55:54.681677 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17b3628b2484ce29ec9926d564bc33c" candidate_uuid: "51a0d14dd9dc423cb658ecb8aaa2ed70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08a418e4c2041a4a79e132d3b2918a9" is_pre_election: true
I20260926 20:55:54.681824 1107 raft_consensus.cc:2454] T b17b3628b2484ce29ec9926d564bc33c P e08a418e4c2041a4a79e132d3b2918a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51a0d14dd9dc423cb658ecb8aaa2ed70 in term 0.
I20260926 20:55:54.681908 1311 leader_election.cc:302] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a0d14dd9dc423cb658ecb8aaa2ed70, 887ce47dc98b45daa48a81e2d854df24; no voters:
I20260926 20:55:54.682078 1445 raft_consensus.cc:2781] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:55:54.682155 1445 raft_consensus.cc:482] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:55:54.682190 1445 raft_consensus.cc:3037] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:54.682945 1445 raft_consensus.cc:504] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a418e4c2041a4a79e132d3b2918a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37949 } } peers { permanent_uuid: "887ce47dc98b45daa48a81e2d854df24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40333 } } peers { permanent_uuid: "51a0d14dd9dc423cb658ecb8aaa2ed70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } }
I20260926 20:55:54.683113 1445 leader_election.cc:288] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70 [CANDIDATE]: Term 1 election: Requested vote from peers e08a418e4c2041a4a79e132d3b2918a9 (127.0.98.1:37949), 887ce47dc98b45daa48a81e2d854df24 (127.0.98.2:40333)
I20260926 20:55:54.683256 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17b3628b2484ce29ec9926d564bc33c" candidate_uuid: "51a0d14dd9dc423cb658ecb8aaa2ed70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08a418e4c2041a4a79e132d3b2918a9"
I20260926 20:55:54.683311 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17b3628b2484ce29ec9926d564bc33c" candidate_uuid: "51a0d14dd9dc423cb658ecb8aaa2ed70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "887ce47dc98b45daa48a81e2d854df24"
I20260926 20:55:54.683362 1107 raft_consensus.cc:3037] T b17b3628b2484ce29ec9926d564bc33c P e08a418e4c2041a4a79e132d3b2918a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:54.683380 1241 raft_consensus.cc:3037] T b17b3628b2484ce29ec9926d564bc33c P 887ce47dc98b45daa48a81e2d854df24 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:54.684324 1107 raft_consensus.cc:2454] T b17b3628b2484ce29ec9926d564bc33c P e08a418e4c2041a4a79e132d3b2918a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a0d14dd9dc423cb658ecb8aaa2ed70 in term 1.
I20260926 20:55:54.684324 1241 raft_consensus.cc:2454] T b17b3628b2484ce29ec9926d564bc33c P 887ce47dc98b45daa48a81e2d854df24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a0d14dd9dc423cb658ecb8aaa2ed70 in term 1.
I20260926 20:55:54.684548 1308 leader_election.cc:302] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a0d14dd9dc423cb658ecb8aaa2ed70, e08a418e4c2041a4a79e132d3b2918a9; no voters:
I20260926 20:55:54.684638 1445 raft_consensus.cc:2781] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:54.684791 1445 raft_consensus.cc:686] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70 [term 1 LEADER]: Becoming Leader. State: Replica: 51a0d14dd9dc423cb658ecb8aaa2ed70, State: Running, Role: LEADER
I20260926 20:55:54.684892 1445 consensus_queue.cc:237] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a418e4c2041a4a79e132d3b2918a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37949 } } peers { permanent_uuid: "887ce47dc98b45daa48a81e2d854df24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40333 } } peers { permanent_uuid: "51a0d14dd9dc423cb658ecb8aaa2ed70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } }
I20260926 20:55:54.685722 962 catalog_manager.cc:5495] T b17b3628b2484ce29ec9926d564bc33c P 51a0d14dd9dc423cb658ecb8aaa2ed70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51a0d14dd9dc423cb658ecb8aaa2ed70 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "51a0d14dd9dc423cb658ecb8aaa2ed70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08a418e4c2041a4a79e132d3b2918a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "887ce47dc98b45daa48a81e2d854df24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51a0d14dd9dc423cb658ecb8aaa2ed70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40399 } health_report { overall_health: HEALTHY } } }
Sep 26 20:55:54 dist-test-slave-fl7g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:55:54.695744 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.687450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46656 (local address 127.0.98.1:37949)
0926 20:55:54.687538 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:54.687540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:54.687605 (+ 65us) server_negotiation.cc:407] Connection header received
0926 20:55:54.687658 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:54.687659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:54.687671 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:54.687729 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:54.688246 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.688808 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.689645 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.689834 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.689899 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.692608 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:54.692618 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:54.692619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:54.692627 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:54.694509 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:54.694927 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:54.694930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:54.694931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:54.694989 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:54.695237 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:54.695237 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:54.695237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:54.695386 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:54.695491 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:54.695573 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 20:55:54.695643 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 20:55:54 dist-test-slave-fl7g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456154, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:55:54 dist-test-slave-fl7g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456154, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:55:54.719681 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.712096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46660 (local address 127.0.98.1:37949)
0926 20:55:54.712179 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:54.712181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:54.712231 (+ 50us) server_negotiation.cc:407] Connection header received
0926 20:55:54.712279 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:54.712279 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:54.712289 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:54.712340 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:54.712990 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.713503 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.714271 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.714528 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.714593 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.716985 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:54.716987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:54.716988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:54.717004 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:54.718654 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:54.719008 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:54.719009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:54.719009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:54.719064 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:54.719231 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:54.719231 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:54.719232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:54.719327 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:54.719409 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:54.719551 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 20:55:54.719614 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 20:55:54.729835 1154 tablet.cc:2369] T b17b3628b2484ce29ec9926d564bc33c P e08a418e4c2041a4a79e132d3b2918a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 20:55:54 dist-test-slave-fl7g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:55:54 dist-test-slave-fl7g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:55:54.744928 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:54.737225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46676 (local address 127.0.98.1:37949)
0926 20:55:54.737339 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:54.737342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:54.737367 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:55:54.737448 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:54.737449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:54.737459 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:54.737522 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:54.738096 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.738639 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.739495 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:54.739707 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:54.739772 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:54.742219 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:54.742221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:54.742221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:54.742228 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:54.743661 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:54.744052 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:54.744054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:54.744054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:54.744121 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:54.744307 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:54.744307 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:54.744307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:54.744509 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:54.744621 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:54.744776 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 20:55:54.744851 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260926 20:55:54.745453 1067 tablet_service.cc:2908] Rejecting scan request for tablet b17b3628b2484ce29ec9926d564bc33c: Uninitialized: clean time has not yet been initialized
I20260926 20:55:54.746444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 1024
I20260926 20:55:54.751808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 1158
I20260926 20:55:54.756661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 1292
I20260926 20:55:54.761117 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 930
2026-09-26T20:55:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2040 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:55:54 dist-test-slave-fl7g krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:55:54 dist-test-slave-fl7g krb5kdc[1462](info): set up 2 sockets
Sep 26 20:55:54 dist-test-slave-fl7g krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260926 20:55:55.853969 392 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 20:55:55.975870 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.122s user 0.000s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 20:55:56.235708 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.260s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260926 20:55:56.479562 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.244s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/test-user.keytab.
W20260926 20:55:57.224076 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.744s user 0.000s sys 0.002s
Sep 26 20:55:57 dist-test-slave-fl7g krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:55:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:55:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260926 20:55:57.374984 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.129s user 0.000s sys 0.003s
I20260926 20:55:57.375401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35169
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:57.458909 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:57.459132 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:57.459156 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:57.460500 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:57.460546 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:55:57.460559 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:57.460598 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:57.460630 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:57.460647 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:57.462201 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35169
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:57.462456 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:57.462730 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:57.467769 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:57.467886 1475 server_base.cc:1034] running on GCE node
W20260926 20:55:57.467746 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:57.467751 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:57.468365 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:57.468711 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:57.469887 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790456157469871 us; error 27 us; skew 500 ppm
Sep 26 20:55:57 dist-test-slave-fl7g krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:57.472640 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:55:57.473109 1475 webserver.cc:460] Webserver started at http://127.0.98.62:40747/ using document root <none> and password file <none>
I20260926 20:55:57.473322 1475 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:57.473366 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:57.473467 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:57.474555 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "360308bb17d0497b9591cad781ed2522"
format_stamp: "Formatted at 2026-09-26 20:55:57 on dist-test-slave-fl7g"
server_key: "22e539fea47a0e5a62b75b2fbcac0bcb"
server_key_iv: "80a5b3e4a3ddb5b4ebdc463cb59a98e3"
server_key_version: "encryptionkey@0"
I20260926 20:55:57.474874 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "360308bb17d0497b9591cad781ed2522"
format_stamp: "Formatted at 2026-09-26 20:55:57 on dist-test-slave-fl7g"
server_key: "22e539fea47a0e5a62b75b2fbcac0bcb"
server_key_iv: "80a5b3e4a3ddb5b4ebdc463cb59a98e3"
server_key_version: "encryptionkey@0"
I20260926 20:55:57.476266 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 20:55:57.477233 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:57.477415 1475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:55:57.477515 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "360308bb17d0497b9591cad781ed2522"
format_stamp: "Formatted at 2026-09-26 20:55:57 on dist-test-slave-fl7g"
server_key: "22e539fea47a0e5a62b75b2fbcac0bcb"
server_key_iv: "80a5b3e4a3ddb5b4ebdc463cb59a98e3"
server_key_version: "encryptionkey@0"
I20260926 20:55:57.477588 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:57.507457 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:57.510087 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:57.510222 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:57.514889 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35169
I20260926 20:55:57.514971 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35169 every 8 connection(s)
I20260926 20:55:57.515316 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:55:57.516110 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:57.518488 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522: Bootstrap starting.
I20260926 20:55:57.519143 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:57.519445 1553 log.cc:824] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:57.520161 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522: No bootstrap required, opened a new log
I20260926 20:55:57.520910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 1475
I20260926 20:55:57.520980 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:55:57.521134 392 external_mini_cluster.cc:1428] Setting key 08cf13d48e502470489d7105968621e1
I20260926 20:55:57.522303 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360308bb17d0497b9591cad781ed2522" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35169 } }
I20260926 20:55:57.522466 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:57.522509 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360308bb17d0497b9591cad781ed2522, State: Initialized, Role: FOLLOWER
I20260926 20:55:57.522670 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360308bb17d0497b9591cad781ed2522" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35169 } }
I20260926 20:55:57.522763 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:55:57.522795 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:55:57.522845 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:57.523584 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360308bb17d0497b9591cad781ed2522" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35169 } }
I20260926 20:55:57.523674 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 360308bb17d0497b9591cad781ed2522; no voters:
I20260926 20:55:57.523844 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:55:57.523895 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:57.524070 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522 [term 1 LEADER]: Becoming Leader. State: Replica: 360308bb17d0497b9591cad781ed2522, State: Running, Role: LEADER
I20260926 20:55:57.524111 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:57.525585 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360308bb17d0497b9591cad781ed2522" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35169 } }
I20260926 20:55:57.526226 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 360308bb17d0497b9591cad781ed2522. Latest consensus state: current_term: 1 leader_uuid: "360308bb17d0497b9591cad781ed2522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360308bb17d0497b9591cad781ed2522" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35169 } } }
I20260926 20:55:57.526538 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522 [sys.catalog]: This master's current role is: LEADER
W20260926 20:55:57.526599 1572 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 20:55:57.526371 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "360308bb17d0497b9591cad781ed2522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360308bb17d0497b9591cad781ed2522" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35169 } } }
I20260926 20:55:57.526690 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522 [sys.catalog]: This master's current role is: LEADER
W20260926 20:55:57.526647 1572 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:55:57.526872 1575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:55:57.527033 1575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:55:57.528708 1575 catalog_manager.cc:1347] Generated new cluster ID: 57924a6d7e954a6797d4a046d0723d60
I20260926 20:55:57.528764 1575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:55:57 dist-test-slave-fl7g krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:55:57.534022 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:57.522125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.0.98.62:35169)
0926 20:55:57.522313 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:57.522321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:57.522347 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:55:57.522451 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:57.522456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:57.522528 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:57.522679 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:57.523506 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:57.524892 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:57.526003 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:57.526633 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:57.526743 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:57.529870 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:57.529874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:57.529875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:57.529886 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:57.532345 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:57.532788 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:57.532792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:57.532792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:57.532904 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:57.533124 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:57.533125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:57.533125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:57.533417 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:57.533571 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:57.533694 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 20:55:57.533827 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260926 20:55:57.539182 1575 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:55:57.539719 1575 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:55:57.544906 1575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522: Generated new TSK 0
I20260926 20:55:57.545118 1575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:55:57.572264 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35169
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:57.656881 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:57.657091 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:57.657119 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:57.658265 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:57.658303 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:57.658324 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:55:57.658342 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:55:57.659825 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:57.660029 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:57.660211 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:57.665275 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:57.665318 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:57.665342 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:57.665669 1579 server_base.cc:1034] running on GCE node
I20260926 20:55:57.665859 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:57.666087 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:57.667233 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790456157667221 us; error 27 us; skew 500 ppm
Sep 26 20:55:57 dist-test-slave-fl7g krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:57.670054 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:55:57.670590 1579 webserver.cc:460] Webserver started at http://127.0.98.1:36557/ using document root <none> and password file <none>
I20260926 20:55:57.670833 1579 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:57.670902 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:57.671052 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:57.672212 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "d1047bcefef74d35b419d83bee3a8621"
format_stamp: "Formatted at 2026-09-26 20:55:57 on dist-test-slave-fl7g"
server_key: "fd5f6766688eeae3f3528267b70b7bd7"
server_key_iv: "4a155a957498af27cfc86b9c11322d88"
server_key_version: "encryptionkey@0"
I20260926 20:55:57.672622 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d1047bcefef74d35b419d83bee3a8621"
format_stamp: "Formatted at 2026-09-26 20:55:57 on dist-test-slave-fl7g"
server_key: "fd5f6766688eeae3f3528267b70b7bd7"
server_key_iv: "4a155a957498af27cfc86b9c11322d88"
server_key_version: "encryptionkey@0"
I20260926 20:55:57.674136 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:55:57.674917 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:57.675100 1579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:55:57.675222 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "d1047bcefef74d35b419d83bee3a8621"
format_stamp: "Formatted at 2026-09-26 20:55:57 on dist-test-slave-fl7g"
server_key: "fd5f6766688eeae3f3528267b70b7bd7"
server_key_iv: "4a155a957498af27cfc86b9c11322d88"
server_key_version: "encryptionkey@0"
I20260926 20:55:57.675300 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:57.683495 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:57.685983 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:57.686162 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:57.686693 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:57.686762 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:57.686806 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:57.686838 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:57.694355 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36489
I20260926 20:55:57.694432 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36489 every 8 connection(s)
Sep 26 20:55:57 dist-test-slave-fl7g krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:55:57.694866 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:55:57.697497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 1579
I20260926 20:55:57.697649 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:55:57.697854 392 external_mini_cluster.cc:1428] Setting key d7754d4c42a4c0c9d978a84d9d2151fd
I20260926 20:55:57.698244 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:57.687290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54061 (local address 127.0.98.62:35169)
0926 20:55:57.687368 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:57.687371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:57.687626 (+ 255us) server_negotiation.cc:407] Connection header received
0926 20:55:57.688310 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:57.688313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:57.688332 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:57.688438 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:57.689515 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:57.690379 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:57.692160 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:57.692527 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:57.692613 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:57.695190 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:57.695192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:57.695192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:57.695204 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:57.696643 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:57.697116 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:57.697117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:57.697118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:57.697181 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:57.697415 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:57.697415 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:57.697416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:57.697585 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:57.697674 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:57.698050 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 20:55:57.698156 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:55:57.698441 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:57.687313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35169 (local address 127.0.98.1:54061)
0926 20:55:57.687489 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:57.687531 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:57.688156 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:57.688594 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:57.688600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:57.689033 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:57.689353 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:57.689361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:57.690520 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:57.690521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:57.692038 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:57.692043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:57.692640 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:57.692641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:57.692762 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:57.693436 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:57.693445 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:57.695037 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:57.696753 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:57.696759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:57.696762 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:57.697018 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:57.697244 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:57.697244 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:57.697244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:57.697335 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:57.697703 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:57.697712 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:57.697864 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 20:55:57.697997 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 20:55:57.698185 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260926 20:55:57.698988 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35169
I20260926 20:55:57.699090 1710 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:57.699334 1710 heartbeater.cc:511] Master 127.0.98.62:35169 requested a full tablet report, sending...
I20260926 20:55:57.699929 1506 ts_manager.cc:194] Registered new tserver with Master: d1047bcefef74d35b419d83bee3a8621 (127.0.98.1:36489)
I20260926 20:55:57.700855 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54061
I20260926 20:55:57.706903 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:57.700071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33411 (local address 127.0.98.62:35169)
0926 20:55:57.700111 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:57.700113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:57.700210 (+ 97us) server_negotiation.cc:407] Connection header received
0926 20:55:57.700287 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:57.700288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:57.700299 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:57.700363 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:57.700794 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:57.701469 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:57.702618 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:57.702960 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:57.703007 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:57.704214 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:57.704215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:57.704215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:57.704222 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:57.705731 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:57.706141 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:57.706142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:57.706142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:57.706204 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:57.706414 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:57.706414 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:57.706414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:57.706516 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:57.706597 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:57.706765 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 20:55:57.706836 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":19}
I20260926 20:55:57.706983 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:57.700011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35169 (local address 127.0.98.1:33411)
0926 20:55:57.700145 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:57.700157 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:57.700221 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:57.700367 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:57.700368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:57.700627 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:57.700706 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:57.700709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:57.701583 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:57.701584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:57.702495 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:57.702497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:57.703095 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:57.703096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:57.703168 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:57.703640 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:57.703646 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:57.704113 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:57.705824 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:57.705825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:57.705825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:57.706058 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:57.706268 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:57.706269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:57.706269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:57.706343 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:57.706616 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:57.706617 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:57.706670 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 20:55:57.706751 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 20:55:57.706856 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:55:57.727249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35169
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:57.809924 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:57.810200 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:57.810243 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:57.811501 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:57.811594 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:57.811630 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:55:57.811671 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:55:57.813352 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:57.813687 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:57.813938 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:57.819531 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:57.819639 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:57.819525 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:57.819741 1719 server_base.cc:1034] running on GCE node
I20260926 20:55:57.820093 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:57.820333 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:57.821534 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790456157821513 us; error 37 us; skew 500 ppm
Sep 26 20:55:57 dist-test-slave-fl7g krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:57.824546 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:55:57.825093 1719 webserver.cc:460] Webserver started at http://127.0.98.2:39467/ using document root <none> and password file <none>
I20260926 20:55:57.825335 1719 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:57.825407 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:57.825564 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:57.826681 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "d544db5d093a4f20945630bb04ddd2b6"
format_stamp: "Formatted at 2026-09-26 20:55:57 on dist-test-slave-fl7g"
server_key: "480e059edc41cc254118ff551a090794"
server_key_iv: "f6ec655f3d4686784cd95c28cc98e94c"
server_key_version: "encryptionkey@0"
I20260926 20:55:57.827069 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d544db5d093a4f20945630bb04ddd2b6"
format_stamp: "Formatted at 2026-09-26 20:55:57 on dist-test-slave-fl7g"
server_key: "480e059edc41cc254118ff551a090794"
server_key_iv: "f6ec655f3d4686784cd95c28cc98e94c"
server_key_version: "encryptionkey@0"
I20260926 20:55:57.828574 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:55:57.829401 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:57.829586 1719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:55:57.829694 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "d544db5d093a4f20945630bb04ddd2b6"
format_stamp: "Formatted at 2026-09-26 20:55:57 on dist-test-slave-fl7g"
server_key: "480e059edc41cc254118ff551a090794"
server_key_iv: "f6ec655f3d4686784cd95c28cc98e94c"
server_key_version: "encryptionkey@0"
I20260926 20:55:57.829775 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:57.861438 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:57.864066 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:57.864189 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:57.864801 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:57.864845 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:57.864867 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:57.864885 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:55:57 dist-test-slave-fl7g krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:55:57.871930 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43331
I20260926 20:55:57.872011 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43331 every 8 connection(s)
I20260926 20:55:57.872329 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:55:57.872745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 1719
I20260926 20:55:57.872838 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:55:57.873030 392 external_mini_cluster.cc:1428] Setting key 62242fb4f66be60f6b32d57f30232dbe
I20260926 20:55:57.875829 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:57.865456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47441 (local address 127.0.98.62:35169)
0926 20:55:57.865531 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:57.865533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:57.865818 (+ 285us) server_negotiation.cc:407] Connection header received
0926 20:55:57.866495 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:57.866498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:57.866512 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:57.866589 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:57.867520 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:57.868375 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:57.869486 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:57.869705 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:57.869789 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:57.872440 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:57.872442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:57.872442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:57.872451 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:57.874218 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:57.874742 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:57.874742 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:57.874743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:57.874799 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:57.875082 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:57.875082 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:57.875082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:57.875186 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:57.875236 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:57.875617 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 20:55:57.875712 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:55:57.875897 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:57.865452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35169 (local address 127.0.98.2:47441)
0926 20:55:57.865674 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:57.865716 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:57.866371 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:57.866690 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:57.866696 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:57.867121 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:57.867423 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:57.867431 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:57.868519 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:57.868521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:57.869379 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:57.869382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:57.869797 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:57.869798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:57.869943 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:57.870533 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:57.870552 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:57.872306 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:57.874306 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:57.874313 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:57.874315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:57.874664 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:57.874867 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:57.874868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:57.874869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:57.875017 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:57.875251 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:57.875256 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:57.875445 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 20:55:57.875571 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 20:55:57.875706 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":151,"threads_started":1}
I20260926 20:55:57.876471 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35169
I20260926 20:55:57.876565 1850 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:57.876754 1850 heartbeater.cc:511] Master 127.0.98.62:35169 requested a full tablet report, sending...
I20260926 20:55:57.877153 1506 ts_manager.cc:194] Registered new tserver with Master: d544db5d093a4f20945630bb04ddd2b6 (127.0.98.2:43331)
I20260926 20:55:57.877727 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47441
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 20:55:57.883625 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:57.877857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50729 (local address 127.0.98.62:35169)
0926 20:55:57.877896 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:57.877899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:57.877907 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:55:57.877927 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:57.877928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:57.877938 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:57.877999 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:57.878359 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:57.878835 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:57.879651 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:57.879857 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:57.879934 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:57.880976 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:57.880978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:57.880978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:57.880986 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:57.882405 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:57.882762 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:57.882763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:57.882764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:57.882826 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:57.883023 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:57.883024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:57.883024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:57.883205 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:57.883284 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:57.883443 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 20:55:57.883524 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 20:55:57.883638 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:57.877597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35169 (local address 127.0.98.2:50729)
0926 20:55:57.877747 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:57.877759 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:57.877811 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:57.878005 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:57.878006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:57.878220 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:57.878271 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:57.878273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:57.878968 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:57.878969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:57.879557 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:57.879559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:57.879951 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:57.879953 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:57.880041 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:57.880579 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:57.880584 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:57.880880 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:57.882498 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:57.882499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:57.882500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:57.882687 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:57.882904 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:57.882905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:57.882905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:57.882971 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:57.883286 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:57.883287 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:57.883334 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 20:55:57.883417 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 20:55:57.883526 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:55:57.904098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35169
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:57.986055 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:57.986286 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:57.986312 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:57.987658 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:57.987706 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:57.987727 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:55:57.987746 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:55:57.989535 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:57.989782 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:57.990022 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:57.995069 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:57.995097 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:57.995049 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:57.995225 1859 server_base.cc:1034] running on GCE node
I20260926 20:55:57.995431 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:57.995682 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:57.996840 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790456157996826 us; error 31 us; skew 500 ppm
Sep 26 20:55:57 dist-test-slave-fl7g krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:57.999481 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:55:57.999958 1859 webserver.cc:460] Webserver started at http://127.0.98.3:45479/ using document root <none> and password file <none>
I20260926 20:55:58.000193 1859 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:58.000263 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:58.000466 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:58.001572 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "1b7133718aa14f8dacb7cac24d7a9118"
format_stamp: "Formatted at 2026-09-26 20:55:57 on dist-test-slave-fl7g"
server_key: "417dfddb79ce8a992b9be4489607578e"
server_key_iv: "606bc46e78b2184885b5914526cd5ce0"
server_key_version: "encryptionkey@0"
I20260926 20:55:58.001963 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1b7133718aa14f8dacb7cac24d7a9118"
format_stamp: "Formatted at 2026-09-26 20:55:57 on dist-test-slave-fl7g"
server_key: "417dfddb79ce8a992b9be4489607578e"
server_key_iv: "606bc46e78b2184885b5914526cd5ce0"
server_key_version: "encryptionkey@0"
I20260926 20:55:58.003422 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:55:58.004209 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:58.004390 1859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:55:58.004571 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "1b7133718aa14f8dacb7cac24d7a9118"
format_stamp: "Formatted at 2026-09-26 20:55:57 on dist-test-slave-fl7g"
server_key: "417dfddb79ce8a992b9be4489607578e"
server_key_iv: "606bc46e78b2184885b5914526cd5ce0"
server_key_version: "encryptionkey@0"
I20260926 20:55:58.004657 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:58.013437 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:58.015314 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:58.015487 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:58.016058 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:58.016124 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:58.016188 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:58.016235 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:55:58 dist-test-slave-fl7g krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:55:58.023737 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40587
I20260926 20:55:58.023808 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40587 every 8 connection(s)
I20260926 20:55:58.024140 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:55:58.027364 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.016683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52267 (local address 127.0.98.62:35169)
0926 20:55:58.016729 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:58.016731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:58.016934 (+ 203us) server_negotiation.cc:407] Connection header received
0926 20:55:58.017463 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:58.017464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:58.017481 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:58.017563 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:58.018269 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.019115 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.020645 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.020865 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.020957 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.023969 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:58.023971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:58.023971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:58.023980 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:58.025671 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:58.026182 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:58.026183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:58.026183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:58.026238 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:58.026601 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:58.026601 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:58.026602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:58.026752 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:58.026826 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:58.027205 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 20:55:58.027293 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:55:58.027488 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.016670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35169 (local address 127.0.98.3:52267)
0926 20:55:58.016821 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:58.016856 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:58.017354 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:58.017664 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:58.017668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:58.017987 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:58.018188 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.018193 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.019213 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.019215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.020543 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.020547 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.020985 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.020986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.021124 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.021720 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:58.021729 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:58.023841 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:58.025781 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:58.025788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:58.025789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:58.026100 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:58.026325 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:58.026325 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:58.026326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:58.026536 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:58.026851 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:58.026859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:58.027039 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 20:55:58.027159 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 20:55:58.027308 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260926 20:55:58.028149 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35169
I20260926 20:55:58.028239 1990 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:58.028457 1990 heartbeater.cc:511] Master 127.0.98.62:35169 requested a full tablet report, sending...
I20260926 20:55:58.028891 1507 ts_manager.cc:194] Registered new tserver with Master: 1b7133718aa14f8dacb7cac24d7a9118 (127.0.98.3:40587)
I20260926 20:55:58.029222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 1859
I20260926 20:55:58.029333 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:55:58.029397 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52267
I20260926 20:55:58.029517 392 external_mini_cluster.cc:1428] Setting key 6b57d7f153e4a0b301b1ce62bc2d7da4
I20260926 20:55:58.030694 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:55:58.036778 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.029498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35169 (local address 127.0.98.3:51827)
0926 20:55:58.029678 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:58.029689 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:58.029776 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:58.029925 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:58.029926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:58.030269 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:58.030351 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.030354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.031340 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.031341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.031878 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.031880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.032515 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.032516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.032598 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.033123 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:58.033129 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:58.033635 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:58.035484 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:58.035494 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:58.035494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:58.035745 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:58.036062 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:58.036063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:58.036063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:58.036124 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:58.036485 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:58.036486 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:58.036533 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 20:55:58.036589 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:55:58.036697 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":106,"threads_started":1}
I20260926 20:55:58.036880 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.029580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51827 (local address 127.0.98.62:35169)
0926 20:55:58.029636 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:58.029638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:58.029824 (+ 186us) server_negotiation.cc:407] Connection header received
0926 20:55:58.029845 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:58.029846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:58.029856 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:58.029919 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:58.030433 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.031221 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.031979 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.032336 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.032446 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.033743 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:58.033745 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:58.033745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:58.033773 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:58.035363 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:58.035838 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:58.035840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:58.035840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:58.035933 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:58.036207 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:58.036208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:58.036209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:58.036365 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:58.036483 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:58.036636 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 20:55:58.036731 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 20:55:58 dist-test-slave-fl7g krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456158, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:55:58 dist-test-slave-fl7g krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456158, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:55:58.066692 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.057910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.0.98.62:35169)
0926 20:55:58.057976 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:58.057978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:58.057990 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:55:58.058022 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:58.058023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:58.058058 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:58.058160 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:58.059119 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.060021 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.061047 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.061266 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.061346 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.064010 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:58.064011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:58.064012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:58.064020 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:58.065487 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:58.065862 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:58.065864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:58.065864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:58.065925 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:58.066144 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:58.066145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:58.066145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:58.066266 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:58.066377 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:58.066492 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 20:55:58.066580 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:55:58.068732 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:55:58.070477 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:55:58.078557 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.073342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43331 (local address 127.0.0.1:34662)
0926 20:55:58.073468 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:58.073511 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:58.073630 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:58.074221 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:58.074224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:58.074227 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:58.074307 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.074309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.075805 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.075806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.077023 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.077025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.078161 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.078163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.078277 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.078290 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 20:55:58.078387 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 20:55:58.078450 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260926 20:55:58.078567 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.073728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34662 (local address 127.0.98.2:43331)
0926 20:55:58.074001 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:58.074007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:58.074020 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:55:58.074066 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:58.074068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:58.074086 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:58.074395 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:58.074505 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.075707 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.077117 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.077942 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.078026 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.078120 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:58.078404 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 20:55:58.078458 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":106,"threads_started":1}
I20260926 20:55:58.079046 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.073656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36489 (local address 127.0.0.1:38156)
0926 20:55:58.074260 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:58.074269 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:58.074353 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:58.074537 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:58.074538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:58.074540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:58.074592 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.074594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.075459 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.075460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.076863 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.076866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.078813 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.078814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.078874 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.078878 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:55:58.078922 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 20:55:58.078963 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":83,"threads_started":1}
I20260926 20:55:58.079306 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.074022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38156 (local address 127.0.98.1:36489)
0926 20:55:58.074213 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:58.074218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:58.074345 (+ 127us) server_negotiation.cc:407] Connection header received
0926 20:55:58.074420 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:58.074422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:58.074437 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:58.074540 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:58.074648 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.075356 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.076968 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.077931 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.078035 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.078117 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:58.079167 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0926 20:55:58.079215 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":75,"threads_started":1}
I20260926 20:55:58.079526 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 8c15ba0a9f414a2aa0a390a96f36d4dd (DEFAULT_TABLE table=test-table [id=f2c79fabc3e44517a88a97d7d1af9535]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:58.079965 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c15ba0a9f414a2aa0a390a96f36d4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:58.079986 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.073543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40587 (local address 127.0.0.1:33062)
0926 20:55:58.073776 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:58.073787 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:58.073828 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:58.074271 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:58.074271 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:58.074274 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:58.074308 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.074309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.077144 (+ 2835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.077145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.077938 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.077939 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.079683 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.079684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.079766 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.079772 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:55:58.079835 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 20:55:58.079878 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":95,"threads_started":1}
I20260926 20:55:58.080022 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.073816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.0.98.3:40587)
0926 20:55:58.074001 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:58.074007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:58.074020 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:55:58.074066 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:58.074068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:58.074086 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:58.074190 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:58.074397 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.075697 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.078032 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.079596 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.079674 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.079757 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:58.079872 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 20:55:58.079922 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260926 20:55:58.080870 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 8c15ba0a9f414a2aa0a390a96f36d4dd (DEFAULT_TABLE table=test-table [id=f2c79fabc3e44517a88a97d7d1af9535]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:58.081274 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c15ba0a9f414a2aa0a390a96f36d4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:58.083639 2013 tablet_bootstrap.cc:492] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6: Bootstrap starting.
I20260926 20:55:58.084264 2013 tablet_bootstrap.cc:654] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:58.084496 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 8c15ba0a9f414a2aa0a390a96f36d4dd (DEFAULT_TABLE table=test-table [id=f2c79fabc3e44517a88a97d7d1af9535]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:58.084688 2013 log.cc:824] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:58.084875 2014 tablet_bootstrap.cc:492] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118: Bootstrap starting.
I20260926 20:55:58.084895 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c15ba0a9f414a2aa0a390a96f36d4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:58.085525 2013 tablet_bootstrap.cc:492] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6: No bootstrap required, opened a new log
I20260926 20:55:58.085561 2014 tablet_bootstrap.cc:654] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:58.085614 2013 ts_tablet_manager.cc:1397] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:55:58.085855 2014 log.cc:824] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:58.086680 2014 tablet_bootstrap.cc:492] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118: No bootstrap required, opened a new log
I20260926 20:55:58.086751 2014 ts_tablet_manager.cc:1397] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:55:58.087637 2013 raft_consensus.cc:348] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.087814 2013 raft_consensus.cc:374] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:58.087849 2013 raft_consensus.cc:729] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d544db5d093a4f20945630bb04ddd2b6, State: Initialized, Role: FOLLOWER
I20260926 20:55:58.087972 2013 consensus_queue.cc:260] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.088261 2013 ts_tablet_manager.cc:1428] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:55:58.088387 1850 heartbeater.cc:503] Master 127.0.98.62:35169 was elected leader, sending a full tablet report...
I20260926 20:55:58.088744 2018 tablet_bootstrap.cc:492] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621: Bootstrap starting.
I20260926 20:55:58.089043 2014 raft_consensus.cc:348] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.089190 2014 raft_consensus.cc:374] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:58.089212 2014 raft_consensus.cc:729] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b7133718aa14f8dacb7cac24d7a9118, State: Initialized, Role: FOLLOWER
I20260926 20:55:58.089332 2014 consensus_queue.cc:260] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.089589 2014 ts_tablet_manager.cc:1428] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:55:58.089721 1990 heartbeater.cc:503] Master 127.0.98.62:35169 was elected leader, sending a full tablet report...
I20260926 20:55:58.089728 2018 tablet_bootstrap.cc:654] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:58.090102 2018 log.cc:824] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:58.090925 2018 tablet_bootstrap.cc:492] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621: No bootstrap required, opened a new log
I20260926 20:55:58.090993 2018 ts_tablet_manager.cc:1397] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:55:58.092571 2018 raft_consensus.cc:348] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.092706 2018 raft_consensus.cc:374] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:58.092729 2018 raft_consensus.cc:729] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1047bcefef74d35b419d83bee3a8621, State: Initialized, Role: FOLLOWER
I20260926 20:55:58.092876 2018 consensus_queue.cc:260] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.093147 2018 ts_tablet_manager.cc:1428] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:55:58.093209 1710 heartbeater.cc:503] Master 127.0.98.62:35169 was elected leader, sending a full tablet report...
W20260926 20:55:58.123767 1851 tablet.cc:2369] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:55:58.141927 2019 raft_consensus.cc:482] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:55:58.142138 2019 raft_consensus.cc:504] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.142601 2019 leader_election.cc:288] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d544db5d093a4f20945630bb04ddd2b6 (127.0.98.2:43331), d1047bcefef74d35b419d83bee3a8621 (127.0.98.1:36489)
I20260926 20:55:58.145529 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.142812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43331 (local address 127.0.98.3:43243)
0926 20:55:58.142879 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:58.142893 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:58.142975 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:58.143152 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:58.143153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:58.143155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:58.143224 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.143226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.143911 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.143912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.144708 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.144710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.145257 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.145258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.145344 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.145347 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:55:58.145412 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:55:58.145455 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 20:55:58.145529 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.142935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43243 (local address 127.0.98.2:43331)
0926 20:55:58.142999 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:58.143001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:58.143019 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:55:58.143050 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:58.143051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:58.143064 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:58.143145 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:58.143318 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.143810 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.144810 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.145168 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.145258 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.145269 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:58.145429 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 20:55:58.145463 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:55:58.145880 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c15ba0a9f414a2aa0a390a96f36d4dd" candidate_uuid: "1b7133718aa14f8dacb7cac24d7a9118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d544db5d093a4f20945630bb04ddd2b6" is_pre_election: true
I20260926 20:55:58.146083 1804 raft_consensus.cc:2454] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b7133718aa14f8dacb7cac24d7a9118 in term 0.
I20260926 20:55:58.146107 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.143128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33611 (local address 127.0.98.1:36489)
0926 20:55:58.143181 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:58.143184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:58.143268 (+ 84us) server_negotiation.cc:407] Connection header received
0926 20:55:58.143348 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:58.143349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:58.143369 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:58.143455 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:58.143632 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.144136 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.145222 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.145676 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.145758 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.145768 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:58.145994 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 20:55:58.146035 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 20:55:58.146136 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.143029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36489 (local address 127.0.98.3:33611)
0926 20:55:58.143211 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:58.143222 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:58.143278 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:58.143466 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:58.143467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:58.143469 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:58.143540 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.143542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.144232 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.144233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.145123 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.145125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.145816 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.145817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.145889 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.145892 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:55:58.145985 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 20:55:58.146025 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":115,"threads_started":1}
I20260926 20:55:58.146410 1876 leader_election.cc:302] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b7133718aa14f8dacb7cac24d7a9118, d544db5d093a4f20945630bb04ddd2b6; no voters:
I20260926 20:55:58.146447 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c15ba0a9f414a2aa0a390a96f36d4dd" candidate_uuid: "1b7133718aa14f8dacb7cac24d7a9118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1047bcefef74d35b419d83bee3a8621" is_pre_election: true
I20260926 20:55:58.146565 2019 raft_consensus.cc:2781] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:55:58.146610 1664 raft_consensus.cc:2454] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b7133718aa14f8dacb7cac24d7a9118 in term 0.
I20260926 20:55:58.146637 2019 raft_consensus.cc:482] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:55:58.146667 2019 raft_consensus.cc:3037] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:58.147522 2019 raft_consensus.cc:504] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.147667 2019 leader_election.cc:288] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [CANDIDATE]: Term 1 election: Requested vote from peers d544db5d093a4f20945630bb04ddd2b6 (127.0.98.2:43331), d1047bcefef74d35b419d83bee3a8621 (127.0.98.1:36489)
I20260926 20:55:58.147871 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c15ba0a9f414a2aa0a390a96f36d4dd" candidate_uuid: "1b7133718aa14f8dacb7cac24d7a9118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1047bcefef74d35b419d83bee3a8621"
I20260926 20:55:58.147866 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c15ba0a9f414a2aa0a390a96f36d4dd" candidate_uuid: "1b7133718aa14f8dacb7cac24d7a9118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d544db5d093a4f20945630bb04ddd2b6"
I20260926 20:55:58.147958 1664 raft_consensus.cc:3037] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:58.147958 1804 raft_consensus.cc:3037] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:58.148974 1804 raft_consensus.cc:2454] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b7133718aa14f8dacb7cac24d7a9118 in term 1.
I20260926 20:55:58.148974 1664 raft_consensus.cc:2454] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b7133718aa14f8dacb7cac24d7a9118 in term 1.
I20260926 20:55:58.149227 1876 leader_election.cc:302] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b7133718aa14f8dacb7cac24d7a9118, d544db5d093a4f20945630bb04ddd2b6; no voters:
I20260926 20:55:58.149349 2019 raft_consensus.cc:2781] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:58.149520 2019 raft_consensus.cc:686] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [term 1 LEADER]: Becoming Leader. State: Replica: 1b7133718aa14f8dacb7cac24d7a9118, State: Running, Role: LEADER
I20260926 20:55:58.149606 2019 consensus_queue.cc:237] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.150902 1507 catalog_manager.cc:5495] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b7133718aa14f8dacb7cac24d7a9118 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1b7133718aa14f8dacb7cac24d7a9118" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } health_report { overall_health: HEALTHY } } }
I20260926 20:55:58.176544 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.173785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33068 (local address 127.0.98.3:40587)
0926 20:55:58.173848 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:58.173850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:58.173865 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:55:58.173997 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:58.173998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:58.174027 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:58.174114 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:58.174276 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.174747 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.175618 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.175815 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.175893 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.176145 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:58.176260 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:58.176357 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:58.176397 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 20:55:58.176446 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:55:58.178530 1804 raft_consensus.cc:1261] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6 [term 1 FOLLOWER]: Refusing update from remote peer 1b7133718aa14f8dacb7cac24d7a9118: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:58.178572 1664 raft_consensus.cc:1261] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621 [term 1 FOLLOWER]: Refusing update from remote peer 1b7133718aa14f8dacb7cac24d7a9118: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:58.178893 2019 consensus_queue.cc:1035] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:58.179050 2023 consensus_queue.cc:1035] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:58.182241 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333708423896399872 to 7333708423782666240. Current Snapshot: MvccSnapshot[applied={T|T < 7333708423896399872}]
I20260926 20:55:58.182317 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333708423896399872 to 7333708423782666240. Current Snapshot: MvccSnapshot[applied={T|T < 7333708423896399872}]
I20260926 20:55:58.186496 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:55:58.186621 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:55:58.187500 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522: Sending DeleteTablet for 3 replicas of tablet 8c15ba0a9f414a2aa0a390a96f36d4dd
I20260926 20:55:58.187865 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c15ba0a9f414a2aa0a390a96f36d4dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:55:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34662
I20260926 20:55:58.187896 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c15ba0a9f414a2aa0a390a96f36d4dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:55:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33062
I20260926 20:55:58.188159 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c15ba0a9f414a2aa0a390a96f36d4dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:55:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38156
I20260926 20:55:58.188392 2038 tablet_replica.cc:330] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621: stopping tablet replica
I20260926 20:55:58.188515 2038 raft_consensus.cc:2229] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:55:58.188552 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 20:55:58.188656 2038 raft_consensus.cc:2258] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 20:55:58.188714 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:55:58.189004 2037 tablet_replica.cc:330] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118: stopping tablet replica
I20260926 20:55:58.189292 2038 ts_tablet_manager.cc:1905] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:55:58.190201 1924 tablet_service.cc:1467] Processing CreateTablet for tablet cffdb382720f409abf1ace2afe3f0f17 (DEFAULT_TABLE table=test-table [id=3428d08ca3644de1b95cf36a880f14f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:58.190330 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cffdb382720f409abf1ace2afe3f0f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:58.190711 1784 tablet_service.cc:1467] Processing CreateTablet for tablet cffdb382720f409abf1ace2afe3f0f17 (DEFAULT_TABLE table=test-table [id=3428d08ca3644de1b95cf36a880f14f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:58.190838 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cffdb382720f409abf1ace2afe3f0f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:58.191718 2014 tablet_bootstrap.cc:492] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118: Bootstrap starting.
I20260926 20:55:58.192000 2013 tablet_bootstrap.cc:492] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6: Bootstrap starting.
I20260926 20:55:58.192094 1644 tablet_service.cc:1467] Processing CreateTablet for tablet cffdb382720f409abf1ace2afe3f0f17 (DEFAULT_TABLE table=test-table [id=3428d08ca3644de1b95cf36a880f14f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:58.192137 2014 tablet_bootstrap.cc:654] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:58.192206 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cffdb382720f409abf1ace2afe3f0f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:58.192488 2013 tablet_bootstrap.cc:654] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:58.192847 2036 tablet_replica.cc:330] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6: stopping tablet replica
I20260926 20:55:58.192975 2036 raft_consensus.cc:2229] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:55:58.193120 2036 raft_consensus.cc:2258] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:55:58.193245 2037 raft_consensus.cc:2229] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:55:58.193408 2037 raft_consensus.cc:2258] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:55:58.193712 2018 tablet_bootstrap.cc:492] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621: Bootstrap starting.
I20260926 20:55:58.193814 2036 ts_tablet_manager.cc:1905] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:55:58.194145 2018 tablet_bootstrap.cc:654] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:58.194526 2037 ts_tablet_manager.cc:1905] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:55:58.195228 2038 ts_tablet_manager.cc:1918] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:55:58.195322 2038 log.cc:1196] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/wal/wals/8c15ba0a9f414a2aa0a390a96f36d4dd
I20260926 20:55:58.195603 2038 ts_tablet_manager.cc:1939] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d1047bcefef74d35b419d83bee3a8621: Deleting consensus metadata
I20260926 20:55:58.196059 1492 catalog_manager.cc:4841] TS d1047bcefef74d35b419d83bee3a8621 (127.0.98.1:36489): tablet 8c15ba0a9f414a2aa0a390a96f36d4dd (table test-table [id=f2c79fabc3e44517a88a97d7d1af9535]) successfully deleted
I20260926 20:55:58.196130 2018 tablet_bootstrap.cc:492] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621: No bootstrap required, opened a new log
I20260926 20:55:58.196195 2018 ts_tablet_manager.cc:1397] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:55:58.196355 2018 raft_consensus.cc:348] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.196460 2018 raft_consensus.cc:374] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:58.196488 2018 raft_consensus.cc:729] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1047bcefef74d35b419d83bee3a8621, State: Initialized, Role: FOLLOWER
I20260926 20:55:58.196542 2018 consensus_queue.cc:260] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.196652 2018 ts_tablet_manager.cc:1428] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:55:58.196977 2037 ts_tablet_manager.cc:1918] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:55:58.197100 2037 log.cc:1196] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/8c15ba0a9f414a2aa0a390a96f36d4dd
I20260926 20:55:58.197408 2037 ts_tablet_manager.cc:1939] T 8c15ba0a9f414a2aa0a390a96f36d4dd P 1b7133718aa14f8dacb7cac24d7a9118: Deleting consensus metadata
I20260926 20:55:58.197858 1493 catalog_manager.cc:4841] TS 1b7133718aa14f8dacb7cac24d7a9118 (127.0.98.3:40587): tablet 8c15ba0a9f414a2aa0a390a96f36d4dd (table test-table [id=f2c79fabc3e44517a88a97d7d1af9535]) successfully deleted
I20260926 20:55:58.197880 2036 ts_tablet_manager.cc:1918] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:55:58.197971 2036 log.cc:1196] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/8c15ba0a9f414a2aa0a390a96f36d4dd
I20260926 20:55:58.198252 2036 ts_tablet_manager.cc:1939] T 8c15ba0a9f414a2aa0a390a96f36d4dd P d544db5d093a4f20945630bb04ddd2b6: Deleting consensus metadata
I20260926 20:55:58.199131 1492 catalog_manager.cc:4841] TS d544db5d093a4f20945630bb04ddd2b6 (127.0.98.2:43331): tablet 8c15ba0a9f414a2aa0a390a96f36d4dd (table test-table [id=f2c79fabc3e44517a88a97d7d1af9535]) successfully deleted
I20260926 20:55:58.200685 2014 tablet_bootstrap.cc:492] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118: No bootstrap required, opened a new log
I20260926 20:55:58.200881 2013 tablet_bootstrap.cc:492] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6: No bootstrap required, opened a new log
I20260926 20:55:58.200748 2014 ts_tablet_manager.cc:1397] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260926 20:55:58.200953 2013 ts_tablet_manager.cc:1397] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260926 20:55:58.201124 2013 raft_consensus.cc:348] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.201116 2014 raft_consensus.cc:348] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.201215 2014 raft_consensus.cc:374] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:58.201215 2013 raft_consensus.cc:374] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:58.201251 2013 raft_consensus.cc:729] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d544db5d093a4f20945630bb04ddd2b6, State: Initialized, Role: FOLLOWER
I20260926 20:55:58.201251 2014 raft_consensus.cc:729] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b7133718aa14f8dacb7cac24d7a9118, State: Initialized, Role: FOLLOWER
I20260926 20:55:58.201303 2013 consensus_queue.cc:260] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.201311 2014 consensus_queue.cc:260] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.201414 2013 ts_tablet_manager.cc:1428] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 20:55:58.201421 2014 ts_tablet_manager.cc:1428] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
W20260926 20:55:58.274755 1991 tablet.cc:2369] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:55:58.287096 2017 raft_consensus.cc:482] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:55:58.287202 2017 raft_consensus.cc:504] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.287568 2017 leader_election.cc:288] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1047bcefef74d35b419d83bee3a8621 (127.0.98.1:36489), 1b7133718aa14f8dacb7cac24d7a9118 (127.0.98.3:40587)
I20260926 20:55:58.291446 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.287795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36489 (local address 127.0.98.2:34843)
0926 20:55:58.287847 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:58.287860 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:58.287901 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:58.288217 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:58.288218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:58.288219 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:58.288300 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.288302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.289080 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.289081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.290449 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.290452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.291198 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.291199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.291270 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.291273 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:55:58.291320 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 20:55:58.291352 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":41}
I20260926 20:55:58.291456 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.287795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40587 (local address 127.0.98.2:32823)
0926 20:55:58.287977 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:58.287988 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:58.288091 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:58.288262 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:58.288263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:58.288265 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:58.288372 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.288374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.289359 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.289360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.290466 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.290468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.291112 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.291113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.291196 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.291200 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:55:58.291284 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:55:58.291334 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":52,"thread_start_us":75,"threads_started":1}
I20260926 20:55:58.291450 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.287862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32823 (local address 127.0.98.3:40587)
0926 20:55:58.287908 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:58.287910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:58.288094 (+ 184us) server_negotiation.cc:407] Connection header received
0926 20:55:58.288172 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:58.288173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:58.288184 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:58.288229 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:58.288481 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.289264 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.290590 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.291018 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.291119 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.291132 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:58.291336 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 20:55:58.291381 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:55:58.291839 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.288023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34843 (local address 127.0.98.1:36489)
0926 20:55:58.288095 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:58.288097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:58.288105 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:55:58.288134 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:58.288135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:58.288146 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:58.288198 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:58.288378 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.288968 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.290570 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.291122 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.291205 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.291217 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:58.291723 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 20:55:58.291776 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:55:58.291955 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdb382720f409abf1ace2afe3f0f17" candidate_uuid: "d544db5d093a4f20945630bb04ddd2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b7133718aa14f8dacb7cac24d7a9118" is_pre_election: true
I20260926 20:55:58.292065 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdb382720f409abf1ace2afe3f0f17" candidate_uuid: "d544db5d093a4f20945630bb04ddd2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1047bcefef74d35b419d83bee3a8621" is_pre_election: true
I20260926 20:55:58.292122 2031 raft_consensus.cc:482] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:55:58.292148 1664 raft_consensus.cc:2454] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d544db5d093a4f20945630bb04ddd2b6 in term 0.
I20260926 20:55:58.292181 2031 raft_consensus.cc:504] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.292330 2031 leader_election.cc:288] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1047bcefef74d35b419d83bee3a8621 (127.0.98.1:36489), d544db5d093a4f20945630bb04ddd2b6 (127.0.98.2:43331)
I20260926 20:55:58.292510 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdb382720f409abf1ace2afe3f0f17" candidate_uuid: "1b7133718aa14f8dacb7cac24d7a9118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d544db5d093a4f20945630bb04ddd2b6" is_pre_election: true
I20260926 20:55:58.292536 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdb382720f409abf1ace2afe3f0f17" candidate_uuid: "1b7133718aa14f8dacb7cac24d7a9118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1047bcefef74d35b419d83bee3a8621" is_pre_election: true
I20260926 20:55:58.292567 1944 raft_consensus.cc:2454] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d544db5d093a4f20945630bb04ddd2b6 in term 0.
I20260926 20:55:58.292598 1664 raft_consensus.cc:2454] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b7133718aa14f8dacb7cac24d7a9118 in term 0.
I20260926 20:55:58.292598 1804 raft_consensus.cc:2454] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b7133718aa14f8dacb7cac24d7a9118 in term 0.
I20260926 20:55:58.292742 1876 leader_election.cc:302] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b7133718aa14f8dacb7cac24d7a9118, d544db5d093a4f20945630bb04ddd2b6; no voters:
I20260926 20:55:58.292865 1736 leader_election.cc:302] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1047bcefef74d35b419d83bee3a8621, d544db5d093a4f20945630bb04ddd2b6; no voters:
I20260926 20:55:58.293016 2031 raft_consensus.cc:2781] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:55:58.293031 2017 raft_consensus.cc:2781] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:55:58.293057 2031 raft_consensus.cc:482] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:55:58.293073 2017 raft_consensus.cc:482] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:55:58.293077 2031 raft_consensus.cc:3037] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:58.293094 2017 raft_consensus.cc:3037] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:58.293774 2017 raft_consensus.cc:504] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.293885 2017 leader_election.cc:288] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [CANDIDATE]: Term 1 election: Requested vote from peers d1047bcefef74d35b419d83bee3a8621 (127.0.98.1:36489), 1b7133718aa14f8dacb7cac24d7a9118 (127.0.98.3:40587)
I20260926 20:55:58.293941 2031 raft_consensus.cc:504] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.294063 2031 leader_election.cc:288] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [CANDIDATE]: Term 1 election: Requested vote from peers d1047bcefef74d35b419d83bee3a8621 (127.0.98.1:36489), d544db5d093a4f20945630bb04ddd2b6 (127.0.98.2:43331)
I20260926 20:55:58.294356 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdb382720f409abf1ace2afe3f0f17" candidate_uuid: "1b7133718aa14f8dacb7cac24d7a9118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1047bcefef74d35b419d83bee3a8621"
I20260926 20:55:58.294360 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdb382720f409abf1ace2afe3f0f17" candidate_uuid: "1b7133718aa14f8dacb7cac24d7a9118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d544db5d093a4f20945630bb04ddd2b6"
I20260926 20:55:58.294436 1664 raft_consensus.cc:3037] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:58.294454 1804 raft_consensus.cc:2379] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b7133718aa14f8dacb7cac24d7a9118 in current term 1: Already voted for candidate d544db5d093a4f20945630bb04ddd2b6 in this term.
I20260926 20:55:58.294523 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdb382720f409abf1ace2afe3f0f17" candidate_uuid: "d544db5d093a4f20945630bb04ddd2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b7133718aa14f8dacb7cac24d7a9118"
I20260926 20:55:58.294610 1944 raft_consensus.cc:2379] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d544db5d093a4f20945630bb04ddd2b6 in current term 1: Already voted for candidate 1b7133718aa14f8dacb7cac24d7a9118 in this term.
I20260926 20:55:58.294870 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdb382720f409abf1ace2afe3f0f17" candidate_uuid: "d544db5d093a4f20945630bb04ddd2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1047bcefef74d35b419d83bee3a8621"
I20260926 20:55:58.295135 1736 leader_election.cc:302] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d544db5d093a4f20945630bb04ddd2b6; no voters: 1b7133718aa14f8dacb7cac24d7a9118, d1047bcefef74d35b419d83bee3a8621
I20260926 20:55:58.295241 2017 raft_consensus.cc:2726] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 20:55:58.295348 1664 raft_consensus.cc:2454] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b7133718aa14f8dacb7cac24d7a9118 in term 1.
I20260926 20:55:58.295522 1876 leader_election.cc:302] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b7133718aa14f8dacb7cac24d7a9118, d1047bcefef74d35b419d83bee3a8621; no voters: d544db5d093a4f20945630bb04ddd2b6
I20260926 20:55:58.295648 2031 raft_consensus.cc:2781] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:58.295717 2031 raft_consensus.cc:686] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [term 1 LEADER]: Becoming Leader. State: Replica: 1b7133718aa14f8dacb7cac24d7a9118, State: Running, Role: LEADER
I20260926 20:55:58.295761 2031 consensus_queue.cc:237] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.296383 1507 catalog_manager.cc:5495] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b7133718aa14f8dacb7cac24d7a9118 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1b7133718aa14f8dacb7cac24d7a9118" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } health_report { overall_health: HEALTHY } } }
Sep 26 20:55:58 dist-test-slave-fl7g krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:55:58.340592 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.333321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49802 (local address 127.0.98.62:35169)
0926 20:55:58.333383 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:58.333385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:58.333507 (+ 122us) server_negotiation.cc:407] Connection header received
0926 20:55:58.333580 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:58.333581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:58.333596 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:58.333679 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:58.334180 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.334678 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.335363 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.335543 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.335598 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.337877 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:58.337878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:58.337879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:58.337886 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:58.339344 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:58.339779 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:58.339781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:58.339781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:58.339848 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:58.340039 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:58.340040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:58.340040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:58.340157 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:58.340211 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:58.340331 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 20:55:58.340469 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 20:55:58.340732 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.333258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35169 (local address 127.0.0.1:49802)
0926 20:55:58.333438 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:58.333448 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:58.333509 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:58.333694 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:58.333695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:58.334058 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:58.334109 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.334112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.334751 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.334751 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.335289 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.335291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.335609 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.335610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.335651 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.336249 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:58.336253 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:58.337789 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:58.339425 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:58.339431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:58.339433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:58.339653 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:58.339926 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:58.339926 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:58.339927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:58.339983 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:58.340213 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:58.340218 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:58.340263 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 20:55:58.340315 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 20:55:58.340529 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":104,"thread_start_us":104,"threads_started":1}
I20260926 20:55:58.343168 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49802:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:55:58.343372 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:55:58.344867 1924 tablet_service.cc:1467] Processing CreateTablet for tablet b00f9ddce8b84b99920e95eab35c8065 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f7a0d0453fa48d6b7428cebb03e9ac4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:55:58.344976 1784 tablet_service.cc:1467] Processing CreateTablet for tablet b00f9ddce8b84b99920e95eab35c8065 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f7a0d0453fa48d6b7428cebb03e9ac4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:55:58.345073 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00f9ddce8b84b99920e95eab35c8065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:58.345163 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00f9ddce8b84b99920e95eab35c8065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:58.346390 2014 tablet_bootstrap.cc:492] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118: Bootstrap starting.
I20260926 20:55:58.346598 2013 tablet_bootstrap.cc:492] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6: Bootstrap starting.
I20260926 20:55:58.346642 1644 tablet_service.cc:1467] Processing CreateTablet for tablet b00f9ddce8b84b99920e95eab35c8065 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f7a0d0453fa48d6b7428cebb03e9ac4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:55:58.346817 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00f9ddce8b84b99920e95eab35c8065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:58.346985 2014 tablet_bootstrap.cc:654] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:58.347050 2013 tablet_bootstrap.cc:654] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:58.347584 2014 tablet_bootstrap.cc:492] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118: No bootstrap required, opened a new log
I20260926 20:55:58.347658 2014 ts_tablet_manager.cc:1397] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:55:58.347786 2014 raft_consensus.cc:348] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.347884 2014 raft_consensus.cc:374] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:58.347909 2014 raft_consensus.cc:729] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b7133718aa14f8dacb7cac24d7a9118, State: Initialized, Role: FOLLOWER
I20260926 20:55:58.347939 2013 tablet_bootstrap.cc:492] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6: No bootstrap required, opened a new log
I20260926 20:55:58.347980 2013 ts_tablet_manager.cc:1397] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:55:58.347961 2014 consensus_queue.cc:260] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.348065 2014 ts_tablet_manager.cc:1428] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 20:55:58.348129 2013 raft_consensus.cc:348] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.348206 2013 raft_consensus.cc:374] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:58.348229 2013 raft_consensus.cc:729] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d544db5d093a4f20945630bb04ddd2b6, State: Initialized, Role: FOLLOWER
I20260926 20:55:58.348280 2013 consensus_queue.cc:260] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.348380 2013 ts_tablet_manager.cc:1428] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:58.348486 2017 tablet_replica.cc:439] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } } }
I20260926 20:55:58.348582 2017 tablet_replica.cc:442] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:55:58.348687 2031 tablet_replica.cc:439] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } } }
I20260926 20:55:58.348783 2031 tablet_replica.cc:442] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:55:58.348878 2018 tablet_bootstrap.cc:492] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621: Bootstrap starting.
I20260926 20:55:58.349320 2018 tablet_bootstrap.cc:654] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:58.349926 2018 tablet_bootstrap.cc:492] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621: No bootstrap required, opened a new log
I20260926 20:55:58.349970 2018 ts_tablet_manager.cc:1397] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:55:58.350142 2018 raft_consensus.cc:348] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.350203 2018 raft_consensus.cc:374] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:58.350216 2018 raft_consensus.cc:729] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1047bcefef74d35b419d83bee3a8621, State: Initialized, Role: FOLLOWER
I20260926 20:55:58.350270 2018 consensus_queue.cc:260] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.350420 2018 ts_tablet_manager.cc:1428] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:58.350474 2021 tablet_replica.cc:439] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } } }
I20260926 20:55:58.350569 2021 tablet_replica.cc:442] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 20:55:58.445729 1711 tablet.cc:2369] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:55:58.482987 2017 raft_consensus.cc:482] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:55:58.483116 2017 raft_consensus.cc:504] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.483278 2017 leader_election.cc:288] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1047bcefef74d35b419d83bee3a8621 (127.0.98.1:36489), 1b7133718aa14f8dacb7cac24d7a9118 (127.0.98.3:40587)
I20260926 20:55:58.483522 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00f9ddce8b84b99920e95eab35c8065" candidate_uuid: "d544db5d093a4f20945630bb04ddd2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1047bcefef74d35b419d83bee3a8621" is_pre_election: true
I20260926 20:55:58.483665 1664 raft_consensus.cc:2454] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d544db5d093a4f20945630bb04ddd2b6 in term 0.
I20260926 20:55:58.483574 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00f9ddce8b84b99920e95eab35c8065" candidate_uuid: "d544db5d093a4f20945630bb04ddd2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b7133718aa14f8dacb7cac24d7a9118" is_pre_election: true
I20260926 20:55:58.483728 1944 raft_consensus.cc:2454] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d544db5d093a4f20945630bb04ddd2b6 in term 0.
I20260926 20:55:58.483863 1736 leader_election.cc:302] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1047bcefef74d35b419d83bee3a8621, d544db5d093a4f20945630bb04ddd2b6; no voters:
I20260926 20:55:58.483979 2017 raft_consensus.cc:2781] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:55:58.484025 2017 raft_consensus.cc:482] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:55:58.484040 2017 raft_consensus.cc:3037] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:58.484794 2017 raft_consensus.cc:504] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.484938 2017 leader_election.cc:288] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [CANDIDATE]: Term 1 election: Requested vote from peers d1047bcefef74d35b419d83bee3a8621 (127.0.98.1:36489), 1b7133718aa14f8dacb7cac24d7a9118 (127.0.98.3:40587)
I20260926 20:55:58.485126 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00f9ddce8b84b99920e95eab35c8065" candidate_uuid: "d544db5d093a4f20945630bb04ddd2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b7133718aa14f8dacb7cac24d7a9118"
I20260926 20:55:58.485108 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00f9ddce8b84b99920e95eab35c8065" candidate_uuid: "d544db5d093a4f20945630bb04ddd2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1047bcefef74d35b419d83bee3a8621"
I20260926 20:55:58.485215 1664 raft_consensus.cc:3037] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:58.485217 1944 raft_consensus.cc:3037] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:58.485836 1944 raft_consensus.cc:2454] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d544db5d093a4f20945630bb04ddd2b6 in term 1.
I20260926 20:55:58.485909 1664 raft_consensus.cc:2454] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d544db5d093a4f20945630bb04ddd2b6 in term 1.
I20260926 20:55:58.486035 1737 leader_election.cc:302] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b7133718aa14f8dacb7cac24d7a9118, d544db5d093a4f20945630bb04ddd2b6; no voters:
I20260926 20:55:58.486140 2017 raft_consensus.cc:2781] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:58.486299 2017 raft_consensus.cc:686] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [term 1 LEADER]: Becoming Leader. State: Replica: d544db5d093a4f20945630bb04ddd2b6, State: Running, Role: LEADER
I20260926 20:55:58.486382 2017 consensus_queue.cc:237] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } }
I20260926 20:55:58.486867 2053 tablet_replica.cc:439] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6: TxnStatusTablet state changed. Reason: New leader d544db5d093a4f20945630bb04ddd2b6. Latest consensus state: current_term: 1 leader_uuid: "d544db5d093a4f20945630bb04ddd2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } } }
I20260926 20:55:58.486974 2053 tablet_replica.cc:442] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:55:58.487167 2055 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:55:58.487310 2055 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:55:58.487334 1506 catalog_manager.cc:5495] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d544db5d093a4f20945630bb04ddd2b6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d544db5d093a4f20945630bb04ddd2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } health_report { overall_health: UNKNOWN } } }
I20260926 20:55:58.497336 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.494515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43331 (local address 127.0.0.1:49754)
0926 20:55:58.494619 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:58.494633 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:58.494691 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:58.494878 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:58.494879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:58.494880 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:55:58.494942 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.494945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.495653 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.495655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.496270 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.496272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.496748 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.496749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.496817 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.496841 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:55:58.497131 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:55:58.497141 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 20:55:58.497196 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 20:55:58.497250 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 20:55:58.497334 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.494659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.0.98.2:43331)
0926 20:55:58.494708 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:58.494711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:58.494737 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:55:58.494770 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:58.494771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:58.494791 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:58.494872 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:58.495018 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.495566 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.496377 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.496652 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.496746 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.496952 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:58.497061 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:58.497147 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:58.497202 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 20:55:58.497227 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 20:55:58.498728 1664 raft_consensus.cc:1261] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621 [term 1 FOLLOWER]: Refusing update from remote peer d544db5d093a4f20945630bb04ddd2b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:58.498987 1944 raft_consensus.cc:1261] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118 [term 1 FOLLOWER]: Refusing update from remote peer d544db5d093a4f20945630bb04ddd2b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:58.499080 2017 consensus_queue.cc:1035] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:58.499238 2053 consensus_queue.cc:1035] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:58.499960 2031 tablet_replica.cc:439] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118: TxnStatusTablet state changed. Reason: New leader d544db5d093a4f20945630bb04ddd2b6. Latest consensus state: current_term: 1 leader_uuid: "d544db5d093a4f20945630bb04ddd2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } } }
I20260926 20:55:58.500087 2031 tablet_replica.cc:442] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:55:58.500029 2021 tablet_replica.cc:439] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621: TxnStatusTablet state changed. Reason: New leader d544db5d093a4f20945630bb04ddd2b6. Latest consensus state: current_term: 1 leader_uuid: "d544db5d093a4f20945630bb04ddd2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } } }
I20260926 20:55:58.500131 2021 tablet_replica.cc:442] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:55:58.500481 2017 tablet_replica.cc:439] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d544db5d093a4f20945630bb04ddd2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } } }
I20260926 20:55:58.500571 2017 tablet_replica.cc:442] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:55:58.501075 2053 tablet_replica.cc:439] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d544db5d093a4f20945630bb04ddd2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } } }
I20260926 20:55:58.501166 2053 tablet_replica.cc:442] T b00f9ddce8b84b99920e95eab35c8065 P d544db5d093a4f20945630bb04ddd2b6: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:55:58.501178 2021 tablet_replica.cc:439] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d544db5d093a4f20945630bb04ddd2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } } }
I20260926 20:55:58.501263 2021 tablet_replica.cc:442] T b00f9ddce8b84b99920e95eab35c8065 P d1047bcefef74d35b419d83bee3a8621: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:55:58.501942 2031 tablet_replica.cc:439] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d544db5d093a4f20945630bb04ddd2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 } } peers { permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 } } peers { permanent_uuid: "1b7133718aa14f8dacb7cac24d7a9118" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40587 } } }
I20260926 20:55:58.502058 2031 tablet_replica.cc:442] T b00f9ddce8b84b99920e95eab35c8065 P 1b7133718aa14f8dacb7cac24d7a9118: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:55:58.507793 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.505185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45923 (local address 127.0.98.2:43331)
0926 20:55:58.505238 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:58.505241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:58.505269 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:55:58.505335 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:58.505336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:58.505345 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:58.505424 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:58.505556 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.506084 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.506942 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.507173 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.507266 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.507452 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:58.507494 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:58.507578 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:58.507670 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 20:55:58.507693 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:55:58.507790 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.505116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43331 (local address 127.0.98.3:45923)
0926 20:55:58.505212 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:58.505225 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:58.505291 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:58.505427 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:58.505428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:58.505429 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:55:58.505480 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.505482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.506174 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.506175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.506842 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.506844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.507268 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.507268 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.507332 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.507354 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:55:58.507595 (+ 241us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:55:58.507597 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:55:58.507652 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 20:55:58.507701 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 20:55:58.510462 1804 raft_consensus.cc:1261] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [term 1 FOLLOWER]: Refusing update from remote peer 1b7133718aa14f8dacb7cac24d7a9118: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:58.510643 1664 raft_consensus.cc:1261] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621 [term 1 FOLLOWER]: Refusing update from remote peer 1b7133718aa14f8dacb7cac24d7a9118: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:58.510708 2031 consensus_queue.cc:1035] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d544db5d093a4f20945630bb04ddd2b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:58.510854 2023 consensus_queue.cc:1035] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1047bcefef74d35b419d83bee3a8621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:58.512451 2026 mvcc.cc:205] Tried to move back new op lower bound from 7333708425257398272 to 7333708424379969536. Current Snapshot: MvccSnapshot[applied={T|T < 7333708425257398272}]
I20260926 20:55:58.521610 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.517531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37949 (local address 127.0.98.3:40587)
0926 20:55:58.517570 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:58.517572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:58.517876 (+ 304us) server_negotiation.cc:407] Connection header received
0926 20:55:58.518043 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:58.518044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:58.518059 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:58.518090 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:58.518892 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.519670 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.520920 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.521135 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.521188 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.521381 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:58.521409 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:58.521445 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:58.521500 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 20:55:58.521549 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:55:58.521659 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.517485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40587 (local address 127.0.98.2:37949)
0926 20:55:58.517806 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:58.517818 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:58.517906 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:58.518158 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:58.518159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:58.518161 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:55:58.518794 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.518797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.519828 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.519829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.520838 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:58.520841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.521212 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.521213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:58.521281 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.521299 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:55:58.521446 (+ 147us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:55:58.521448 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:55:58.521499 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 20:55:58.521544 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":90,"threads_started":1}
I20260926 20:55:58.529975 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:55:58.530093 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:55:58.530783 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 360308bb17d0497b9591cad781ed2522: Sending DeleteTablet for 3 replicas of tablet cffdb382720f409abf1ace2afe3f0f17
I20260926 20:55:58.531076 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet cffdb382720f409abf1ace2afe3f0f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:55:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33062
I20260926 20:55:58.531104 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet cffdb382720f409abf1ace2afe3f0f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:55:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38156
I20260926 20:55:58.531184 2037 tablet_replica.cc:330] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118: stopping tablet replica
I20260926 20:55:58.531234 2037 raft_consensus.cc:2229] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:55:58.531287 2037 raft_consensus.cc:2258] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:55:58.531586 2037 ts_tablet_manager.cc:1905] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:55:58.531646 2038 tablet_replica.cc:330] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621: stopping tablet replica
I20260926 20:55:58.531728 2038 raft_consensus.cc:2229] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:55:58.531769 2038 raft_consensus.cc:2258] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:55:58.532104 2038 ts_tablet_manager.cc:1905] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:55:58.533373 2037 ts_tablet_manager.cc:1918] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:55:58.533440 2037 log.cc:1196] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/cffdb382720f409abf1ace2afe3f0f17
I20260926 20:55:58.533433 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet cffdb382720f409abf1ace2afe3f0f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:55:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34662
I20260926 20:55:58.533540 2036 tablet_replica.cc:330] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6: stopping tablet replica
I20260926 20:55:58.533571 2038 ts_tablet_manager.cc:1918] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:55:58.533601 2036 raft_consensus.cc:2229] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:55:58.533622 2038 log.cc:1196] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-0/wal/wals/cffdb382720f409abf1ace2afe3f0f17
I20260926 20:55:58.533632 2036 raft_consensus.cc:2258] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:55:58.533679 2037 ts_tablet_manager.cc:1939] T cffdb382720f409abf1ace2afe3f0f17 P 1b7133718aa14f8dacb7cac24d7a9118: Deleting consensus metadata
I20260926 20:55:58.533823 2038 ts_tablet_manager.cc:1939] T cffdb382720f409abf1ace2afe3f0f17 P d1047bcefef74d35b419d83bee3a8621: Deleting consensus metadata
I20260926 20:55:58.533960 2036 ts_tablet_manager.cc:1905] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:55:58.534055 1493 catalog_manager.cc:4841] TS 1b7133718aa14f8dacb7cac24d7a9118 (127.0.98.3:40587): tablet cffdb382720f409abf1ace2afe3f0f17 (table test-table [id=3428d08ca3644de1b95cf36a880f14f9]) successfully deleted
I20260926 20:55:58.534355 1492 catalog_manager.cc:4841] TS d1047bcefef74d35b419d83bee3a8621 (127.0.98.1:36489): tablet cffdb382720f409abf1ace2afe3f0f17 (table test-table [id=3428d08ca3644de1b95cf36a880f14f9]) successfully deleted
I20260926 20:55:58.535389 2036 ts_tablet_manager.cc:1918] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:55:58.535454 2036 log.cc:1196] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/cffdb382720f409abf1ace2afe3f0f17
I20260926 20:55:58.535673 2036 ts_tablet_manager.cc:1939] T cffdb382720f409abf1ace2afe3f0f17 P d544db5d093a4f20945630bb04ddd2b6: Deleting consensus metadata
I20260926 20:55:58.536051 1492 catalog_manager.cc:4841] TS d544db5d093a4f20945630bb04ddd2b6 (127.0.98.2:43331): tablet cffdb382720f409abf1ace2afe3f0f17 (table test-table [id=3428d08ca3644de1b95cf36a880f14f9]) successfully deleted
Sep 26 20:55:58 dist-test-slave-fl7g krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456158, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:55:58.544940 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.531844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54586 (local address 127.0.98.1:36489)
0926 20:55:58.531882 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:58.531884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:58.536606 (+ 4722us) server_negotiation.cc:407] Connection header received
0926 20:55:58.536681 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:58.536682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:58.536696 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:58.536746 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:58.537317 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.537802 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.538557 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.538751 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.538829 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.541912 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:58.541925 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:58.541925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:58.541944 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:58.543636 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:58.544008 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:58.544012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:58.544012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:58.544078 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:58.544291 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:58.544292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:58.544292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:58.544507 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:58.544689 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:58.544738 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 20:55:58.544819 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 20:55:58.545243 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54586
I20260926 20:55:58.552505 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:58.546434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49814 (local address 127.0.98.62:35169)
0926 20:55:58.546480 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:58.546483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:58.546549 (+ 66us) server_negotiation.cc:407] Connection header received
0926 20:55:58.546618 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:58.546619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:58.546629 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:58.546673 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:58.547238 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.547741 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.548558 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:58.548773 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:58.548850 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:58.549990 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:58.549992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:58.549992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:58.550001 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:58.551346 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:58.551693 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:58.551694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:58.551695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:58.551754 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:58.551960 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:58.551961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:58.551962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:58.552082 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:58.552192 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:58.552279 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 20:55:58.552358 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 20:55:58.552788 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49814
I20260926 20:55:58.553436 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 1579
I20260926 20:55:58.560020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 1719
I20260926 20:55:58.565392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 1859
I20260926 20:55:58.571033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 1475
2026-09-26T20:55:58Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3808 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:55:58 dist-test-slave-fl7g krb5kdc[2074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:55:58 dist-test-slave-fl7g krb5kdc[2074](info): set up 2 sockets
Sep 26 20:55:58 dist-test-slave-fl7g krb5kdc[2074](info): commencing operation
krb5kdc: starting...
W20260926 20:55:59.632166 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:55:59 dist-test-slave-fl7g krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:55:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:55:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:55:59.759413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38333
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:59.843865 2087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:59.844146 2087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:59.844187 2087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:59.845515 2087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:59.845606 2087 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:55:59.845633 2087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:59.845654 2087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:59.845675 2087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:59.845705 2087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:59.847971 2087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38333
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:59.848295 2087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:59.848706 2087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:59.854169 2093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:59.854192 2092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:59.854390 2087 server_base.cc:1034] running on GCE node
W20260926 20:55:59.854192 2095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:59.854775 2087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:59.855058 2087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:59.856235 2087 hybrid_clock.cc:648] HybridClock initialized: now 1790456159856215 us; error 31 us; skew 500 ppm
Sep 26 20:55:59 dist-test-slave-fl7g krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:59.864061 2087 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:55:59.864660 2087 webserver.cc:460] Webserver started at http://127.0.98.62:46197/ using document root <none> and password file <none>
I20260926 20:55:59.864919 2087 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:59.864989 2087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:59.865144 2087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:59.866319 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "4c3ff07ee5164e36a26708f88f09ae0c"
format_stamp: "Formatted at 2026-09-26 20:55:59 on dist-test-slave-fl7g"
server_key: "bb1f7a37207fdb29546cfe90694d814d"
server_key_iv: "a82dc111d15d3705a8d9a1fd59571434"
server_key_version: "encryptionkey@0"
I20260926 20:55:59.866717 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "4c3ff07ee5164e36a26708f88f09ae0c"
format_stamp: "Formatted at 2026-09-26 20:55:59 on dist-test-slave-fl7g"
server_key: "bb1f7a37207fdb29546cfe90694d814d"
server_key_iv: "a82dc111d15d3705a8d9a1fd59571434"
server_key_version: "encryptionkey@0"
I20260926 20:55:59.868182 2087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:55:59.869107 2102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:59.869305 2087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:55:59.869408 2087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "4c3ff07ee5164e36a26708f88f09ae0c"
format_stamp: "Formatted at 2026-09-26 20:55:59 on dist-test-slave-fl7g"
server_key: "bb1f7a37207fdb29546cfe90694d814d"
server_key_iv: "a82dc111d15d3705a8d9a1fd59571434"
server_key_version: "encryptionkey@0"
I20260926 20:55:59.869494 2087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:59.879339 2087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:59.881702 2087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:59.881907 2087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:59.886897 2087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38333
I20260926 20:55:59.886938 2164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38333 every 8 connection(s)
I20260926 20:55:59.887288 2087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:55:59.887923 2165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:59.890316 2165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c: Bootstrap starting.
I20260926 20:55:59.890923 2165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:59.891222 2165 log.cc:824] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:59.891950 2165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c: No bootstrap required, opened a new log
I20260926 20:55:59.894423 2165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3ff07ee5164e36a26708f88f09ae0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38333 } }
I20260926 20:55:59.894603 2165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:59.894655 2165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c3ff07ee5164e36a26708f88f09ae0c, State: Initialized, Role: FOLLOWER
I20260926 20:55:59.894793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 2087
I20260926 20:55:59.894826 2165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3ff07ee5164e36a26708f88f09ae0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38333 } }
I20260926 20:55:59.894898 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:55:59.894927 2165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:55:59.894963 2165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:55:59.895011 2165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:59.895100 392 external_mini_cluster.cc:1428] Setting key 9135501d0a55f1037e46d4ba4367ab67
I20260926 20:55:59.895901 2165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3ff07ee5164e36a26708f88f09ae0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38333 } }
I20260926 20:55:59.896044 2165 leader_election.cc:302] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c3ff07ee5164e36a26708f88f09ae0c; no voters:
I20260926 20:55:59.896245 2165 leader_election.cc:288] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:55:59.896281 2170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:59.896521 2170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c [term 1 LEADER]: Becoming Leader. State: Replica: 4c3ff07ee5164e36a26708f88f09ae0c, State: Running, Role: LEADER
I20260926 20:55:59.896626 2165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:59.896667 2170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3ff07ee5164e36a26708f88f09ae0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38333 } }
I20260926 20:55:59.897095 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c3ff07ee5164e36a26708f88f09ae0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3ff07ee5164e36a26708f88f09ae0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38333 } } }
I20260926 20:55:59.897127 2172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c3ff07ee5164e36a26708f88f09ae0c. Latest consensus state: current_term: 1 leader_uuid: "4c3ff07ee5164e36a26708f88f09ae0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3ff07ee5164e36a26708f88f09ae0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38333 } } }
I20260926 20:55:59.897208 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:59.897218 2172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c [sys.catalog]: This master's current role is: LEADER
W20260926 20:55:59.898595 2186 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:55:59.898720 2186 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:55:59.898775 2179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:55:59.898931 2179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:55:59.900580 2179 catalog_manager.cc:1347] Generated new cluster ID: 9d78a07f3d264533a032eb74f934fa74
I20260926 20:55:59.900652 2179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:55:59 dist-test-slave-fl7g krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:55:59.905910 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:59.895712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49398 (local address 127.0.98.62:38333)
0926 20:55:59.895880 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:59.895897 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:59.895925 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:55:59.896058 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:59.896066 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:59.896153 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:59.896327 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:59.897082 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:59.898468 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:59.899411 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:59.900022 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:59.900182 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:59.902516 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:59.902520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:59.902520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:59.902532 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:59.904275 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:59.904731 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:59.904734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:59.904735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:59.904848 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:59.905072 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:59.905073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:59.905073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:59.905320 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:59.905482 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:59.905591 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 20:55:59.905749 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260926 20:55:59.915642 2179 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:55:59.916244 2179 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:55:59.924448 2179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c: Generated new TSK 0
I20260926 20:55:59.924702 2179 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:55:59.954937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38333
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:00.040318 2191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:00.040584 2191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:00.040614 2191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:00.041932 2191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:00.041988 2191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:00.042011 2191 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:56:00.042023 2191 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:56:00.042042 2191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:56:00.043622 2191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:00.043906 2191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:00.044162 2191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:00.049880 2197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:00.049880 2196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:00.049882 2199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:00.049940 2191 server_base.cc:1034] running on GCE node
I20260926 20:56:00.050431 2191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:00.050722 2191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:00.051892 2191 hybrid_clock.cc:648] HybridClock initialized: now 1790456160051870 us; error 41 us; skew 500 ppm
Sep 26 20:56:00 dist-test-slave-fl7g krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:00.054719 2191 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:56:00.055225 2191 webserver.cc:460] Webserver started at http://127.0.98.1:36137/ using document root <none> and password file <none>
I20260926 20:56:00.055490 2191 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:00.055569 2191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:00.055730 2191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:00.056942 2191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "a4d8becac609479ba233629c5d5b8b47"
format_stamp: "Formatted at 2026-09-26 20:56:00 on dist-test-slave-fl7g"
server_key: "a50d54f90137cd987d30acfe2fa5d78a"
server_key_iv: "07645e40212ab2841164d835356358a7"
server_key_version: "encryptionkey@0"
I20260926 20:56:00.057327 2191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a4d8becac609479ba233629c5d5b8b47"
format_stamp: "Formatted at 2026-09-26 20:56:00 on dist-test-slave-fl7g"
server_key: "a50d54f90137cd987d30acfe2fa5d78a"
server_key_iv: "07645e40212ab2841164d835356358a7"
server_key_version: "encryptionkey@0"
I20260926 20:56:00.058733 2191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:56:00.059516 2206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:00.059703 2191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:00.059813 2191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "a4d8becac609479ba233629c5d5b8b47"
format_stamp: "Formatted at 2026-09-26 20:56:00 on dist-test-slave-fl7g"
server_key: "a50d54f90137cd987d30acfe2fa5d78a"
server_key_iv: "07645e40212ab2841164d835356358a7"
server_key_version: "encryptionkey@0"
I20260926 20:56:00.059894 2191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:00.078116 2191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:00.080809 2191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:00.081058 2191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:00.081735 2191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:00.081807 2191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:00.081869 2191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:00.081899 2191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:56:00 dist-test-slave-fl7g krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:00.089597 2191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44501
I20260926 20:56:00.089712 2321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44501 every 8 connection(s)
I20260926 20:56:00.090019 2191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:56:00.090399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 2191
I20260926 20:56:00.090512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:56:00.090737 392 external_mini_cluster.cc:1428] Setting key 8f277ed32b1de7b2571a86d4058ffda0
I20260926 20:56:00.094024 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.082486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54577 (local address 127.0.98.62:38333)
0926 20:56:00.082548 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:00.082551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:00.082823 (+ 272us) server_negotiation.cc:407] Connection header received
0926 20:56:00.083475 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:00.083477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:00.083496 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:00.083575 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:00.084621 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.085229 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.086399 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.086763 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.086845 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.090118 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:00.090120 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:00.090124 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:00.090134 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:00.092221 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:00.092816 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:00.092819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:00.092819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:00.092891 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:00.093165 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:00.093166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:00.093167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:00.093332 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:00.093398 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:00.093837 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 20:56:00.093930 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:56:00.094254 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.082452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38333 (local address 127.0.98.1:54577)
0926 20:56:00.082685 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:00.082727 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:00.083364 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:00.083695 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:00.083721 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:00.084150 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:00.084515 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.084521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.085345 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.085346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.086300 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.086303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.086903 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.086905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.087189 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.087877 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:00.087891 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:00.089979 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:00.092320 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:00.092328 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:00.092330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:00.092731 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:00.092979 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:00.092980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:00.092980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:00.093095 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:00.093432 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:00.093439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:00.093643 (+ 204us) client_negotiation.cc:769] Sending connection context
0926 20:56:00.093764 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 20:56:00.093965 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":122,"threads_started":1}
I20260926 20:56:00.094949 2322 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38333
I20260926 20:56:00.095063 2322 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:00.095260 2322 heartbeater.cc:511] Master 127.0.98.62:38333 requested a full tablet report, sending...
I20260926 20:56:00.095850 2119 ts_manager.cc:194] Registered new tserver with Master: a4d8becac609479ba233629c5d5b8b47 (127.0.98.1:44501)
I20260926 20:56:00.096648 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54577
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 20:56:00.106355 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.100164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59875 (local address 127.0.98.62:38333)
0926 20:56:00.100237 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:00.100239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:00.100338 (+ 99us) server_negotiation.cc:407] Connection header received
0926 20:56:00.100450 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:00.100451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:00.100462 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:00.100537 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:00.100886 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.101394 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.102135 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.102348 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.102426 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.103582 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:00.103583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:00.103584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:00.103593 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:00.105158 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:00.105603 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:00.105604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:00.105604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:00.105667 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:00.105874 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:00.105874 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:00.105875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:00.105983 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:00.106033 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:00.106205 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 20:56:00.106284 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:56:00.106467 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.100066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38333 (local address 127.0.98.1:59875)
0926 20:56:00.100264 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:00.100275 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:00.100357 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:00.100544 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:00.100545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:00.100742 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:00.100805 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.100807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.101494 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.101495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.102013 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.102015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.102423 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.102424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.102513 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.103072 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:00.103078 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:00.103482 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:00.105250 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:00.105251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:00.105252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:00.105505 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:00.105732 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:00.105733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:00.105733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:00.105806 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:00.106045 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:00.106046 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:00.106104 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 20:56:00.106180 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:56:00.106308 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:56:00.120182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38333
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:00.204442 2331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:00.204787 2331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:00.204840 2331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:00.206349 2331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:00.206409 2331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:00.206434 2331 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:56:00.206449 2331 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:56:00.206470 2331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:56:00.208281 2331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:00.208607 2331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:00.208917 2331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:00.214856 2339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:00.215042 2337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:00.215133 2336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:00.215342 2331 server_base.cc:1034] running on GCE node
I20260926 20:56:00.215539 2331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:00.215754 2331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:00.216919 2331 hybrid_clock.cc:648] HybridClock initialized: now 1790456160216901 us; error 34 us; skew 500 ppm
Sep 26 20:56:00 dist-test-slave-fl7g krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:00.219659 2331 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:56:00.220216 2331 webserver.cc:460] Webserver started at http://127.0.98.2:42669/ using document root <none> and password file <none>
I20260926 20:56:00.220521 2331 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:00.220602 2331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:00.220757 2331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:00.221874 2331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "d72172b78b754368ba4a8f3e3f44889a"
format_stamp: "Formatted at 2026-09-26 20:56:00 on dist-test-slave-fl7g"
server_key: "106022c521f098d533f8d0edab195fd7"
server_key_iv: "773e7ebd9fc6d36b1b424e945ddc1b18"
server_key_version: "encryptionkey@0"
I20260926 20:56:00.222256 2331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d72172b78b754368ba4a8f3e3f44889a"
format_stamp: "Formatted at 2026-09-26 20:56:00 on dist-test-slave-fl7g"
server_key: "106022c521f098d533f8d0edab195fd7"
server_key_iv: "773e7ebd9fc6d36b1b424e945ddc1b18"
server_key_version: "encryptionkey@0"
I20260926 20:56:00.223727 2331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:56:00.224608 2346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:00.224833 2331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:00.224936 2331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "d72172b78b754368ba4a8f3e3f44889a"
format_stamp: "Formatted at 2026-09-26 20:56:00 on dist-test-slave-fl7g"
server_key: "106022c521f098d533f8d0edab195fd7"
server_key_iv: "773e7ebd9fc6d36b1b424e945ddc1b18"
server_key_version: "encryptionkey@0"
I20260926 20:56:00.225049 2331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:00.237874 2331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:00.240473 2331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:00.240617 2331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:00.241127 2331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:00.241207 2331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:00.241256 2331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:00.241288 2331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:00.248636 2331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45267
I20260926 20:56:00.248723 2461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45267 every 8 connection(s)
I20260926 20:56:00.249110 2331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:56:00 dist-test-slave-fl7g krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:00.252822 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.241805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56609 (local address 127.0.98.62:38333)
0926 20:56:00.241853 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:00.241855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:00.242119 (+ 264us) server_negotiation.cc:407] Connection header received
0926 20:56:00.242683 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:00.242685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:00.242702 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:00.242788 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:00.243722 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.244591 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.246358 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.246794 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.246871 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.249901 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:00.249903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:00.249904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:00.249912 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:00.251412 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:00.251880 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:00.251881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:00.251881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:00.251937 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:00.252179 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:00.252180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:00.252180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:00.252291 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:00.252343 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:00.252687 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 20:56:00.252754 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:56:00.253031 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.241793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38333 (local address 127.0.98.2:56609)
0926 20:56:00.241984 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:00.242029 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:00.242565 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:00.242849 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:00.242852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:00.243251 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:00.243577 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.243583 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.244701 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.244702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.246238 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.246242 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.246910 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.246911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.247079 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.247784 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:00.247796 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:00.249781 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:00.251517 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:00.251525 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:00.251527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:00.251800 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:00.252003 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:00.252003 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:00.252003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:00.252112 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:00.252347 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:00.252351 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:00.252577 (+ 226us) client_negotiation.cc:769] Sending connection context
0926 20:56:00.252672 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 20:56:00.252808 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":110,"threads_started":1}
I20260926 20:56:00.253548 2462 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38333
I20260926 20:56:00.253643 2462 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:00.253842 2462 heartbeater.cc:511] Master 127.0.98.62:38333 requested a full tablet report, sending...
I20260926 20:56:00.254377 2118 ts_manager.cc:194] Registered new tserver with Master: d72172b78b754368ba4a8f3e3f44889a (127.0.98.2:45267)
I20260926 20:56:00.254945 2118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56609
I20260926 20:56:00.255658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 2331
I20260926 20:56:00.255767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:56:00.255963 392 external_mini_cluster.cc:1428] Setting key 3a4a08ef0bdab2ff19d2fac7813375fd
I20260926 20:56:00.261386 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.254837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58075 (local address 127.0.98.62:38333)
0926 20:56:00.254898 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:00.254900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:00.254978 (+ 78us) server_negotiation.cc:407] Connection header received
0926 20:56:00.255044 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:00.255045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:00.255056 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:00.255129 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:00.255587 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.256266 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.257133 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.257342 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.257428 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.258616 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:00.258618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:00.258618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:00.258625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:00.260145 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:00.260581 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:00.260582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:00.260582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:00.260634 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:00.260844 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:00.260844 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:00.260844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:00.260978 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:00.261026 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:00.261247 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 20:56:00.261317 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:56:00.261482 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.254768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38333 (local address 127.0.98.2:58075)
0926 20:56:00.254912 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:00.254922 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:00.254988 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:00.255139 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:00.255140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:00.255351 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:00.255498 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.255501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.256366 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.256368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.257035 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.257037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.257429 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.257430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.257518 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.258056 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:00.258062 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:00.258525 (+ 463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:00.260235 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:00.260236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:00.260236 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:00.260500 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:00.260707 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:00.260708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:00.260708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:00.260782 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:00.261046 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:00.261047 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:00.261107 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 20:56:00.261189 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:56:00.261319 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":18,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:56:00.284848 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38333
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:00.370651 2471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:00.370971 2471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:00.371026 2471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:00.372567 2471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:00.372660 2471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:00.372702 2471 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:56:00.372725 2471 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:56:00.372783 2471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:56:00.374610 2471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:00.374965 2471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:00.375252 2471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:00.381263 2477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:00.381263 2479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:00.381276 2476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:00.381520 2471 server_base.cc:1034] running on GCE node
I20260926 20:56:00.381862 2471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:00.382169 2471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:00.383348 2471 hybrid_clock.cc:648] HybridClock initialized: now 1790456160383343 us; error 40 us; skew 500 ppm
Sep 26 20:56:00 dist-test-slave-fl7g krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:00.386261 2471 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:56:00.386893 2471 webserver.cc:460] Webserver started at http://127.0.98.3:46493/ using document root <none> and password file <none>
I20260926 20:56:00.387116 2471 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:00.387161 2471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:00.387260 2471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:00.388355 2471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "894967ab7a5b4f09b7309558d7ce327f"
format_stamp: "Formatted at 2026-09-26 20:56:00 on dist-test-slave-fl7g"
server_key: "d5a738d5d218191c4d980acebb3f8116"
server_key_iv: "980619e764fbd307061f11d62d77334f"
server_key_version: "encryptionkey@0"
I20260926 20:56:00.388815 2471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "894967ab7a5b4f09b7309558d7ce327f"
format_stamp: "Formatted at 2026-09-26 20:56:00 on dist-test-slave-fl7g"
server_key: "d5a738d5d218191c4d980acebb3f8116"
server_key_iv: "980619e764fbd307061f11d62d77334f"
server_key_version: "encryptionkey@0"
I20260926 20:56:00.390218 2471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:00.391001 2486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:00.391191 2471 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:00.391280 2471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "894967ab7a5b4f09b7309558d7ce327f"
format_stamp: "Formatted at 2026-09-26 20:56:00 on dist-test-slave-fl7g"
server_key: "d5a738d5d218191c4d980acebb3f8116"
server_key_iv: "980619e764fbd307061f11d62d77334f"
server_key_version: "encryptionkey@0"
I20260926 20:56:00.391342 2471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:00.413442 2471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:00.416019 2471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:00.416147 2471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:00.416745 2471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:00.416800 2471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:00.416826 2471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:00.416838 2471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:00.423907 2471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40625
I20260926 20:56:00.423986 2601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40625 every 8 connection(s)
I20260926 20:56:00.424336 2471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:56:00 dist-test-slave-fl7g krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:00.427947 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.417390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35517 (local address 127.0.98.62:38333)
0926 20:56:00.417440 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:00.417443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:00.417676 (+ 233us) server_negotiation.cc:407] Connection header received
0926 20:56:00.418261 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:00.418262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:00.418276 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:00.418338 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:00.419400 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.420224 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.422068 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.422264 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.422320 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.425091 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:00.425093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:00.425094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:00.425101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:00.426476 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:00.426924 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:00.426925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:00.426925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:00.426978 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:00.427198 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:00.427199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:00.427199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:00.427311 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:00.427365 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:00.427787 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 20:56:00.427879 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:56:00.428094 2495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.417310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38333 (local address 127.0.98.3:35517)
0926 20:56:00.417543 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:00.417583 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:00.418141 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:00.418467 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:00.418473 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:00.418915 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:00.419265 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.419275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.420344 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.420345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.421938 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.421942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.422355 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.422356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.422532 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.423227 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:00.423238 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:00.425013 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:00.426570 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:00.426577 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:00.426579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:00.426839 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:00.427037 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:00.427037 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:00.427038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:00.427144 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:00.427393 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:00.427399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:00.427553 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 20:56:00.427656 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 20:56:00.427871 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":106,"threads_started":1}
I20260926 20:56:00.428735 2602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38333
I20260926 20:56:00.428828 2602 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:00.429028 2602 heartbeater.cc:511] Master 127.0.98.62:38333 requested a full tablet report, sending...
I20260926 20:56:00.429512 2118 ts_manager.cc:194] Registered new tserver with Master: 894967ab7a5b4f09b7309558d7ce327f (127.0.98.3:40625)
I20260926 20:56:00.430070 2118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35517
I20260926 20:56:00.430392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 2471
I20260926 20:56:00.430491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:56:00.430702 392 external_mini_cluster.cc:1428] Setting key ff8d12fff832333667b220e49115ab3c
I20260926 20:56:00.431526 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:56:00.436231 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.429947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35483 (local address 127.0.98.62:38333)
0926 20:56:00.429993 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:00.429995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:00.430100 (+ 105us) server_negotiation.cc:407] Connection header received
0926 20:56:00.430177 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:00.430178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:00.430188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:00.430243 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:00.430632 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.431272 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.431951 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.432168 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.432251 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.433374 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:00.433376 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:00.433376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:00.433384 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:00.434921 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:00.435379 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:00.435380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:00.435381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:00.435447 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:00.435660 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:00.435661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:00.435661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:00.435787 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:00.435865 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:00.436022 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 20:56:00.436111 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:56:00.436244 2608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.429869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38333 (local address 127.0.98.3:35483)
0926 20:56:00.430004 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:00.430015 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:00.430117 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:00.430266 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:00.430267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:00.430470 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:00.430534 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.430537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.431371 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.431372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.431866 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.431867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.432250 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.432250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.432328 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.432924 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:00.432929 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:00.433257 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:00.435042 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:00.435043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:00.435043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:00.435301 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:00.435517 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:00.435518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:00.435518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:00.435579 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:00.435856 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:00.435857 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:00.435904 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 20:56:00.435997 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 20:56:00.436120 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
Sep 26 20:56:00 dist-test-slave-fl7g krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:56:00 dist-test-slave-fl7g krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:00.457250 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.448980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49412 (local address 127.0.98.62:38333)
0926 20:56:00.449049 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:00.449053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:00.449067 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:56:00.449094 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:00.449094 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:00.449103 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:00.449158 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:00.449827 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.450367 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.451241 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.451433 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.451538 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.454622 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:00.454624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:00.454625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:00.454634 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:00.456188 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:00.456633 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:00.456634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:00.456635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:00.456693 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:00.456853 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:00.456853 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:00.456854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:00.456958 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:00.457042 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:00.457114 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 20:56:00.457180 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:56:00.458937 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:56:00.460626 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:56:00.467569 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.463157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40625 (local address 127.0.0.1:45892)
0926 20:56:00.463376 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:00.463460 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:00.463530 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:00.463791 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:00.463795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:00.463797 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:00.463884 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.463887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.465039 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.465040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.466238 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.466241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.467191 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.467192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.467283 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.467297 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 20:56:00.467407 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 20:56:00.467470 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":53,"threads_started":1}
I20260926 20:56:00.467578 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.463307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45892 (local address 127.0.98.3:40625)
0926 20:56:00.463598 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:00.463604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:00.463615 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:56:00.463660 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:00.463662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:00.463677 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:00.463757 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:00.464021 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.464935 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.466342 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.467089 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.467188 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.467256 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:00.467427 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 20:56:00.467473 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":80,"threads_started":1}
I20260926 20:56:00.467993 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.463416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44501 (local address 127.0.0.1:57774)
0926 20:56:00.463845 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:00.463857 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:00.463906 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:00.464248 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:00.464250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:00.464252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:00.464382 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.464384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.465700 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.465701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.466661 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.466664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.467712 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.467713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.467793 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.467798 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:56:00.467855 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 20:56:00.467906 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":82,"thread_start_us":85,"threads_started":1}
I20260926 20:56:00.468047 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.463631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57774 (local address 127.0.98.1:44501)
0926 20:56:00.464031 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:00.464037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:00.464048 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:56:00.464086 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:00.464089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:00.464105 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:00.464181 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:00.464517 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.465602 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.466751 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.467506 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.467596 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.467671 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:00.467888 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 20:56:00.467940 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":125,"threads_started":1}
I20260926 20:56:00.468396 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.463388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49002 (local address 127.0.98.2:45267)
0926 20:56:00.464039 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:00.464043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:00.464052 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:56:00.464087 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:00.464089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:00.464104 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:00.464189 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:00.464427 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.465574 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.466865 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.467559 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.467616 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.468170 (+ 554us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:00.468225 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 20:56:00.468279 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":68,"threads_started":1}
I20260926 20:56:00.467993 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.463193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45267 (local address 127.0.0.1:49002)
0926 20:56:00.463471 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:00.463483 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:00.463592 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:00.464257 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:00.464258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:00.464259 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:00.464325 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.464327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.465666 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.465667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.466773 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.466775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.467794 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.467794 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.467855 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.467859 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:56:00.467897 (+ 38us) client_negotiation.cc:240] Negotiation successful
0926 20:56:00.467924 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":3,"thread_start_us":104,"threads_started":1}
I20260926 20:56:00.468611 2536 tablet_service.cc:1467] Processing CreateTablet for tablet 605c555c03424367a5b719bde1b1c712 (DEFAULT_TABLE table=test-table [id=54b91ad832944fab801c1b39f36f2125]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:00.468726 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 605c555c03424367a5b719bde1b1c712 (DEFAULT_TABLE table=test-table [id=54b91ad832944fab801c1b39f36f2125]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:00.469015 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 605c555c03424367a5b719bde1b1c712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:00.469017 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 605c555c03424367a5b719bde1b1c712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:00.469185 2396 tablet_service.cc:1467] Processing CreateTablet for tablet 605c555c03424367a5b719bde1b1c712 (DEFAULT_TABLE table=test-table [id=54b91ad832944fab801c1b39f36f2125]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:00.469432 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 605c555c03424367a5b719bde1b1c712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:00.472483 2625 tablet_bootstrap.cc:492] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a: Bootstrap starting.
I20260926 20:56:00.472724 2626 tablet_bootstrap.cc:492] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f: Bootstrap starting.
I20260926 20:56:00.472988 2627 tablet_bootstrap.cc:492] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47: Bootstrap starting.
I20260926 20:56:00.473629 2625 tablet_bootstrap.cc:654] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:00.473629 2627 tablet_bootstrap.cc:654] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:00.473990 2625 log.cc:824] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:00.473990 2627 log.cc:824] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:00.474897 2625 tablet_bootstrap.cc:492] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a: No bootstrap required, opened a new log
I20260926 20:56:00.474977 2625 ts_tablet_manager.cc:1397] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 20:56:00.474998 2627 tablet_bootstrap.cc:492] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47: No bootstrap required, opened a new log
I20260926 20:56:00.475086 2627 ts_tablet_manager.cc:1397] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:56:00.475291 2626 tablet_bootstrap.cc:654] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:00.475625 2626 log.cc:824] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:00.476593 2626 tablet_bootstrap.cc:492] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f: No bootstrap required, opened a new log
I20260926 20:56:00.476678 2626 ts_tablet_manager.cc:1397] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 20:56:00.477145 2627 raft_consensus.cc:348] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } }
I20260926 20:56:00.477320 2627 raft_consensus.cc:374] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:00.477365 2627 raft_consensus.cc:729] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d8becac609479ba233629c5d5b8b47, State: Initialized, Role: FOLLOWER
I20260926 20:56:00.477330 2625 raft_consensus.cc:348] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } }
I20260926 20:56:00.477504 2625 raft_consensus.cc:374] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:00.477546 2625 raft_consensus.cc:729] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d72172b78b754368ba4a8f3e3f44889a, State: Initialized, Role: FOLLOWER
I20260926 20:56:00.477517 2627 consensus_queue.cc:260] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } }
I20260926 20:56:00.477702 2625 consensus_queue.cc:260] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } }
I20260926 20:56:00.477829 2627 ts_tablet_manager.cc:1428] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:56:00.477938 2322 heartbeater.cc:503] Master 127.0.98.62:38333 was elected leader, sending a full tablet report...
I20260926 20:56:00.477984 2625 ts_tablet_manager.cc:1428] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:56:00.478092 2462 heartbeater.cc:503] Master 127.0.98.62:38333 was elected leader, sending a full tablet report...
I20260926 20:56:00.478933 2626 raft_consensus.cc:348] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } }
I20260926 20:56:00.479107 2626 raft_consensus.cc:374] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:00.479189 2626 raft_consensus.cc:729] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894967ab7a5b4f09b7309558d7ce327f, State: Initialized, Role: FOLLOWER
I20260926 20:56:00.479317 2626 consensus_queue.cc:260] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } }
I20260926 20:56:00.479609 2626 ts_tablet_manager.cc:1428] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:56:00.479635 2602 heartbeater.cc:503] Master 127.0.98.62:38333 was elected leader, sending a full tablet report...
W20260926 20:56:00.499708 2463 tablet.cc:2369] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:56:00.548864 2633 raft_consensus.cc:482] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:56:00.548995 2633 raft_consensus.cc:504] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } }
I20260926 20:56:00.549388 2633 leader_election.cc:288] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d72172b78b754368ba4a8f3e3f44889a (127.0.98.2:45267), a4d8becac609479ba233629c5d5b8b47 (127.0.98.1:44501)
I20260926 20:56:00.552537 2495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.549692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45267 (local address 127.0.98.3:56061)
0926 20:56:00.549759 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:00.549772 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:00.549827 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:00.550023 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:00.550024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:00.550026 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:00.550107 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.550110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.550854 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.550856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.551670 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.551673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.552223 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.552224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.552305 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.552309 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:56:00.552365 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:56:00.552398 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 20:56:00.552533 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.549803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56061 (local address 127.0.98.2:45267)
0926 20:56:00.549845 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:00.549847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:00.549857 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:56:00.549888 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:00.549889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:00.549901 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:00.549989 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:00.550190 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.550743 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.551770 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.552134 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.552188 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.552197 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:00.552383 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 20:56:00.552460 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:56:00.552605 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.549774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50917 (local address 127.0.98.1:44501)
0926 20:56:00.549836 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:00.549838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:00.550013 (+ 175us) server_negotiation.cc:407] Connection header received
0926 20:56:00.550065 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:00.550066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:00.550079 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:00.550143 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:00.550369 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.550886 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.551822 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.552213 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.552283 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.552296 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:00.552485 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 20:56:00.552529 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:56:00.552877 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.549692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44501 (local address 127.0.98.3:50917)
0926 20:56:00.549923 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:00.549934 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:00.549994 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:00.550176 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:00.550177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:00.550179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:00.550252 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.550254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.550978 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.550978 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.551693 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.551695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.552298 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.552299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.552395 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.552399 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:56:00.552465 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 20:56:00.552501 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":28,"thread_start_us":117,"threads_started":1}
I20260926 20:56:00.552997 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "605c555c03424367a5b719bde1b1c712" candidate_uuid: "894967ab7a5b4f09b7309558d7ce327f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d72172b78b754368ba4a8f3e3f44889a" is_pre_election: true
I20260926 20:56:00.553179 2416 raft_consensus.cc:2454] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 894967ab7a5b4f09b7309558d7ce327f in term 0.
I20260926 20:56:00.553188 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "605c555c03424367a5b719bde1b1c712" candidate_uuid: "894967ab7a5b4f09b7309558d7ce327f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4d8becac609479ba233629c5d5b8b47" is_pre_election: true
I20260926 20:56:00.553308 2276 raft_consensus.cc:2454] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 894967ab7a5b4f09b7309558d7ce327f in term 0.
I20260926 20:56:00.553450 2487 leader_election.cc:302] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 894967ab7a5b4f09b7309558d7ce327f, d72172b78b754368ba4a8f3e3f44889a; no voters:
I20260926 20:56:00.553624 2633 raft_consensus.cc:2781] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:56:00.553712 2633 raft_consensus.cc:482] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:56:00.553750 2633 raft_consensus.cc:3037] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:00.554584 2633 raft_consensus.cc:504] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } }
I20260926 20:56:00.554747 2633 leader_election.cc:288] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [CANDIDATE]: Term 1 election: Requested vote from peers d72172b78b754368ba4a8f3e3f44889a (127.0.98.2:45267), a4d8becac609479ba233629c5d5b8b47 (127.0.98.1:44501)
I20260926 20:56:00.554922 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "605c555c03424367a5b719bde1b1c712" candidate_uuid: "894967ab7a5b4f09b7309558d7ce327f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4d8becac609479ba233629c5d5b8b47"
I20260926 20:56:00.554939 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "605c555c03424367a5b719bde1b1c712" candidate_uuid: "894967ab7a5b4f09b7309558d7ce327f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d72172b78b754368ba4a8f3e3f44889a"
I20260926 20:56:00.555017 2276 raft_consensus.cc:3037] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:00.555017 2416 raft_consensus.cc:3037] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:00.555943 2416 raft_consensus.cc:2454] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 894967ab7a5b4f09b7309558d7ce327f in term 1.
I20260926 20:56:00.555943 2276 raft_consensus.cc:2454] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 894967ab7a5b4f09b7309558d7ce327f in term 1.
I20260926 20:56:00.556175 2488 leader_election.cc:302] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 894967ab7a5b4f09b7309558d7ce327f, a4d8becac609479ba233629c5d5b8b47; no voters:
I20260926 20:56:00.556293 2633 raft_consensus.cc:2781] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:00.556514 2633 raft_consensus.cc:686] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [term 1 LEADER]: Becoming Leader. State: Replica: 894967ab7a5b4f09b7309558d7ce327f, State: Running, Role: LEADER
I20260926 20:56:00.556799 2633 consensus_queue.cc:237] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } }
I20260926 20:56:00.557848 2119 catalog_manager.cc:5495] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f reported cstate change: term changed from 0 to 1, leader changed from <none> to 894967ab7a5b4f09b7309558d7ce327f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "894967ab7a5b4f09b7309558d7ce327f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } health_report { overall_health: UNKNOWN } } }
W20260926 20:56:00.591080 2323 tablet.cc:2369] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 20:56:00 dist-test-slave-fl7g krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:00.612701 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.605359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49428 (local address 127.0.98.62:38333)
0926 20:56:00.605449 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:00.605452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:00.605566 (+ 114us) server_negotiation.cc:407] Connection header received
0926 20:56:00.605689 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:00.605689 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:00.605698 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:00.605764 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:00.606258 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.606749 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.607470 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.607658 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.607731 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.609962 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:00.609964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:00.609964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:00.609971 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:00.611435 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:00.611870 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:00.611871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:00.611871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:00.611923 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:00.612153 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:00.612153 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:00.612154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:00.612256 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:00.612306 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:00.612524 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 20:56:00.612613 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":63}
I20260926 20:56:00.612783 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.605259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38333 (local address 127.0.0.1:49428)
0926 20:56:00.605470 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:00.605485 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:00.605582 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:00.605768 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:00.605769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:00.606102 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:00.606163 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.606166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.606843 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.606844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.607370 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.607372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.607741 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.607742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.607806 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.608336 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:00.608340 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:00.609870 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:00.611534 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:00.611542 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:00.611544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:00.611795 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:00.612010 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:00.612011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:00.612011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:00.612090 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:00.612315 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:00.612321 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:00.612375 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 20:56:00.612503 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 20:56:00.612685 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":67,"thread_start_us":132,"threads_started":1}
I20260926 20:56:00.614886 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49428:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:56:00.615134 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:56:00.616711 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 4817345bc5df4117a39ba2eeaf6dadd8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6284fe27a8294edd980ed86568a19cc0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:56:00.616725 2396 tablet_service.cc:1467] Processing CreateTablet for tablet 4817345bc5df4117a39ba2eeaf6dadd8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6284fe27a8294edd980ed86568a19cc0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:56:00.616916 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4817345bc5df4117a39ba2eeaf6dadd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:00.616921 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4817345bc5df4117a39ba2eeaf6dadd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:00.616937 2536 tablet_service.cc:1467] Processing CreateTablet for tablet 4817345bc5df4117a39ba2eeaf6dadd8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6284fe27a8294edd980ed86568a19cc0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:56:00.617081 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4817345bc5df4117a39ba2eeaf6dadd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:00.618306 2627 tablet_bootstrap.cc:492] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47: Bootstrap starting.
I20260926 20:56:00.618578 2625 tablet_bootstrap.cc:492] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a: Bootstrap starting.
I20260926 20:56:00.618755 2627 tablet_bootstrap.cc:654] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:00.618849 2626 tablet_bootstrap.cc:492] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f: Bootstrap starting.
I20260926 20:56:00.619270 2626 tablet_bootstrap.cc:654] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:00.619298 2625 tablet_bootstrap.cc:654] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:00.619503 2627 tablet_bootstrap.cc:492] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47: No bootstrap required, opened a new log
I20260926 20:56:00.619558 2627 ts_tablet_manager.cc:1397] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:00.619704 2627 raft_consensus.cc:348] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:00.619807 2627 raft_consensus.cc:374] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:00.619832 2627 raft_consensus.cc:729] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d8becac609479ba233629c5d5b8b47, State: Initialized, Role: FOLLOWER
I20260926 20:56:00.619894 2627 consensus_queue.cc:260] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:00.619920 2626 tablet_bootstrap.cc:492] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f: No bootstrap required, opened a new log
I20260926 20:56:00.619961 2626 ts_tablet_manager.cc:1397] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:00.620011 2627 ts_tablet_manager.cc:1428] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:00.620112 2626 raft_consensus.cc:348] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:00.620111 2631 tablet_replica.cc:439] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } }
I20260926 20:56:00.620195 2626 raft_consensus.cc:374] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:00.620201 2631 tablet_replica.cc:442] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:56:00.620221 2626 raft_consensus.cc:729] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894967ab7a5b4f09b7309558d7ce327f, State: Initialized, Role: FOLLOWER
I20260926 20:56:00.620270 2626 consensus_queue.cc:260] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:00.620363 2626 ts_tablet_manager.cc:1428] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:00.620477 2625 tablet_bootstrap.cc:492] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a: No bootstrap required, opened a new log
I20260926 20:56:00.620496 2633 tablet_replica.cc:439] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } }
I20260926 20:56:00.620553 2625 ts_tablet_manager.cc:1397] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:56:00.620572 2633 tablet_replica.cc:442] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:56:00.620725 2625 raft_consensus.cc:348] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:00.620805 2625 raft_consensus.cc:374] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:00.620831 2625 raft_consensus.cc:729] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d72172b78b754368ba4a8f3e3f44889a, State: Initialized, Role: FOLLOWER
I20260926 20:56:00.620884 2625 consensus_queue.cc:260] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:00.620973 2625 ts_tablet_manager.cc:1428] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:00.621114 2632 tablet_replica.cc:439] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } }
I20260926 20:56:00.621245 2632 tablet_replica.cc:442] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 20:56:00.675164 2603 tablet.cc:2369] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:56:00.695603 2633 raft_consensus.cc:482] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:56:00.695724 2633 raft_consensus.cc:504] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:00.696044 2633 leader_election.cc:288] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4d8becac609479ba233629c5d5b8b47 (127.0.98.1:44501), d72172b78b754368ba4a8f3e3f44889a (127.0.98.2:45267)
I20260926 20:56:00.696309 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4817345bc5df4117a39ba2eeaf6dadd8" candidate_uuid: "894967ab7a5b4f09b7309558d7ce327f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4d8becac609479ba233629c5d5b8b47" is_pre_election: true
I20260926 20:56:00.696442 2276 raft_consensus.cc:2454] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 894967ab7a5b4f09b7309558d7ce327f in term 0.
I20260926 20:56:00.696386 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4817345bc5df4117a39ba2eeaf6dadd8" candidate_uuid: "894967ab7a5b4f09b7309558d7ce327f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d72172b78b754368ba4a8f3e3f44889a" is_pre_election: true
I20260926 20:56:00.696556 2416 raft_consensus.cc:2454] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 894967ab7a5b4f09b7309558d7ce327f in term 0.
I20260926 20:56:00.696758 2487 leader_election.cc:302] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 894967ab7a5b4f09b7309558d7ce327f, d72172b78b754368ba4a8f3e3f44889a; no voters:
I20260926 20:56:00.696872 2633 raft_consensus.cc:2781] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:56:00.696921 2633 raft_consensus.cc:482] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:56:00.696954 2633 raft_consensus.cc:3037] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:00.697741 2633 raft_consensus.cc:504] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:00.697891 2633 leader_election.cc:288] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [CANDIDATE]: Term 1 election: Requested vote from peers a4d8becac609479ba233629c5d5b8b47 (127.0.98.1:44501), d72172b78b754368ba4a8f3e3f44889a (127.0.98.2:45267)
I20260926 20:56:00.698035 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4817345bc5df4117a39ba2eeaf6dadd8" candidate_uuid: "894967ab7a5b4f09b7309558d7ce327f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4d8becac609479ba233629c5d5b8b47"
I20260926 20:56:00.698120 2276 raft_consensus.cc:3037] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:00.698113 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4817345bc5df4117a39ba2eeaf6dadd8" candidate_uuid: "894967ab7a5b4f09b7309558d7ce327f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d72172b78b754368ba4a8f3e3f44889a"
I20260926 20:56:00.698199 2416 raft_consensus.cc:3037] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:00.698922 2416 raft_consensus.cc:2454] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 894967ab7a5b4f09b7309558d7ce327f in term 1.
I20260926 20:56:00.698922 2276 raft_consensus.cc:2454] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 894967ab7a5b4f09b7309558d7ce327f in term 1.
I20260926 20:56:00.699129 2488 leader_election.cc:302] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 894967ab7a5b4f09b7309558d7ce327f, a4d8becac609479ba233629c5d5b8b47; no voters:
I20260926 20:56:00.699267 2633 raft_consensus.cc:2781] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:00.699337 2633 raft_consensus.cc:686] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [term 1 LEADER]: Becoming Leader. State: Replica: 894967ab7a5b4f09b7309558d7ce327f, State: Running, Role: LEADER
I20260926 20:56:00.699394 2633 consensus_queue.cc:237] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:00.699743 2635 tablet_replica.cc:439] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f: TxnStatusTablet state changed. Reason: New leader 894967ab7a5b4f09b7309558d7ce327f. Latest consensus state: current_term: 1 leader_uuid: "894967ab7a5b4f09b7309558d7ce327f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } }
I20260926 20:56:00.699852 2635 tablet_replica.cc:442] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:56:00.700090 2647 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:56:00.700109 2119 catalog_manager.cc:5495] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f reported cstate change: term changed from 0 to 1, leader changed from <none> to 894967ab7a5b4f09b7309558d7ce327f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "894967ab7a5b4f09b7309558d7ce327f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } health_report { overall_health: HEALTHY } } }
I20260926 20:56:00.700227 2647 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:56:00.715726 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.712630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40625 (local address 127.0.0.1:45902)
0926 20:56:00.712799 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:00.712810 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:00.712863 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:00.713045 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:00.713046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:00.713047 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:56:00.713092 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.713095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.714021 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.714022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.714657 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.714659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.715141 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.715142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.715196 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.715213 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:56:00.715540 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:56:00.715550 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 20:56:00.715607 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 20:56:00.715661 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260926 20:56:00.715742 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.712722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.0.98.3:40625)
0926 20:56:00.712772 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:00.712774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:00.712848 (+ 74us) server_negotiation.cc:407] Connection header received
0926 20:56:00.712927 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:00.712928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:00.712950 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:00.713024 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:00.713154 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.713915 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.714747 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.715050 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.715126 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.715308 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:00.715444 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:00.715528 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:00.715620 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 20:56:00.715652 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:56:00.717813 2416 raft_consensus.cc:1261] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a [term 1 FOLLOWER]: Refusing update from remote peer 894967ab7a5b4f09b7309558d7ce327f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:00.717844 2276 raft_consensus.cc:1261] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47 [term 1 FOLLOWER]: Refusing update from remote peer 894967ab7a5b4f09b7309558d7ce327f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:00.718164 2635 consensus_queue.cc:1035] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:00.718315 2633 consensus_queue.cc:1035] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:00.719651 2632 tablet_replica.cc:439] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a: TxnStatusTablet state changed. Reason: New leader 894967ab7a5b4f09b7309558d7ce327f. Latest consensus state: current_term: 1 leader_uuid: "894967ab7a5b4f09b7309558d7ce327f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } }
I20260926 20:56:00.719796 2632 tablet_replica.cc:442] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:56:00.720902 2631 tablet_replica.cc:439] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47: TxnStatusTablet state changed. Reason: New leader 894967ab7a5b4f09b7309558d7ce327f. Latest consensus state: current_term: 1 leader_uuid: "894967ab7a5b4f09b7309558d7ce327f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } }
I20260926 20:56:00.721019 2631 tablet_replica.cc:442] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:56:00.721598 2633 tablet_replica.cc:439] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "894967ab7a5b4f09b7309558d7ce327f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } }
I20260926 20:56:00.721700 2633 tablet_replica.cc:442] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:56:00.721793 2633 tablet_replica.cc:439] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "894967ab7a5b4f09b7309558d7ce327f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } }
I20260926 20:56:00.721839 2633 tablet_replica.cc:442] T 4817345bc5df4117a39ba2eeaf6dadd8 P 894967ab7a5b4f09b7309558d7ce327f: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:56:00.721894 2650 mvcc.cc:205] Tried to move back new op lower bound from 7333708434297425920 to 7333708434225131520. Current Snapshot: MvccSnapshot[applied={T|T < 7333708434297425920}]
I20260926 20:56:00.722030 2632 tablet_replica.cc:439] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "894967ab7a5b4f09b7309558d7ce327f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } }
I20260926 20:56:00.722110 2632 tablet_replica.cc:442] T 4817345bc5df4117a39ba2eeaf6dadd8 P d72172b78b754368ba4a8f3e3f44889a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:56:00.722759 2631 tablet_replica.cc:439] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "894967ab7a5b4f09b7309558d7ce327f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } } }
I20260926 20:56:00.722841 2631 tablet_replica.cc:442] T 4817345bc5df4117a39ba2eeaf6dadd8 P a4d8becac609479ba233629c5d5b8b47: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:56:00.723481 2653 mvcc.cc:205] Tried to move back new op lower bound from 7333708434297425920 to 7333708434225131520. Current Snapshot: MvccSnapshot[applied={T|T < 7333708434297425920}]
I20260926 20:56:00.726758 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.724391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45914 (local address 127.0.98.3:40625)
0926 20:56:00.724484 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:00.724486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:00.724495 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:56:00.724550 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:00.724551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:00.724559 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:00.724627 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:00.724785 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.725266 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.726081 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.726293 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.726368 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.726527 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:00.726560 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:00.726608 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:00.726655 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 20:56:00.726678 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:56:00.731819 2608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.729517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40625 (local address 127.0.98.3:33589)
0926 20:56:00.729573 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:00.729584 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:00.729629 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:00.729744 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:00.729744 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:00.729745 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:56:00.729810 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.729812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.730425 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.730426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.731024 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:00.731025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.731367 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.731368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:00.731425 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.731442 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:56:00.731643 (+ 201us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:56:00.731644 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:56:00.731692 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 20:56:00.731745 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":2}
I20260926 20:56:00.731819 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:00.729569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33589 (local address 127.0.98.3:40625)
0926 20:56:00.729607 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:00.729608 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:00.729616 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:56:00.729679 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:00.729679 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:00.729689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:00.729741 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:00.729879 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.730343 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.731110 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:00.731297 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:00.731370 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:00.731546 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:00.731582 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:00.731643 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:00.731694 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 20:56:00.731733 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":22}
I20260926 20:56:00.734258 2416 raft_consensus.cc:1261] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a [term 1 FOLLOWER]: Refusing update from remote peer 894967ab7a5b4f09b7309558d7ce327f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:00.734258 2276 raft_consensus.cc:1261] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47 [term 1 FOLLOWER]: Refusing update from remote peer 894967ab7a5b4f09b7309558d7ce327f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:00.734490 2635 consensus_queue.cc:1035] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:00.734609 2633 consensus_queue.cc:1035] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:00.749734 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:00.749866 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:00.750674 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c: Sending DeleteTablet for 3 replicas of tablet 605c555c03424367a5b719bde1b1c712
I20260926 20:56:00.750988 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet 605c555c03424367a5b719bde1b1c712 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49002
I20260926 20:56:00.751049 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 605c555c03424367a5b719bde1b1c712 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57774
I20260926 20:56:00.751299 2666 tablet_replica.cc:330] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47: stopping tablet replica
I20260926 20:56:00.751272 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:56:00.751398 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:56:00.751441 2666 raft_consensus.cc:2229] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:00.751509 2666 raft_consensus.cc:2258] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:00.751613 2536 tablet_service.cc:1514] Processing DeleteTablet for tablet 605c555c03424367a5b719bde1b1c712 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45892
I20260926 20:56:00.752022 2666 ts_tablet_manager.cc:1905] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:00.752060 2667 tablet_replica.cc:330] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f: stopping tablet replica
I20260926 20:56:00.752241 2667 raft_consensus.cc:2229] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:56:00.752396 2667 raft_consensus.cc:2258] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:00.752902 2667 ts_tablet_manager.cc:1905] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:00.753017 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 4395759135b84b5faf476339ce2cf6c1 (DEFAULT_TABLE table=test-table [id=1d6300439bc740daa85189f2beb249ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:00.753226 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4395759135b84b5faf476339ce2cf6c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:00.754042 2666 ts_tablet_manager.cc:1918] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:56:00.754122 2666 log.cc:1196] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/wal/wals/605c555c03424367a5b719bde1b1c712
I20260926 20:56:00.754463 2666 ts_tablet_manager.cc:1939] T 605c555c03424367a5b719bde1b1c712 P a4d8becac609479ba233629c5d5b8b47: Deleting consensus metadata
I20260926 20:56:00.754546 2536 tablet_service.cc:1467] Processing CreateTablet for tablet 4395759135b84b5faf476339ce2cf6c1 (DEFAULT_TABLE table=test-table [id=1d6300439bc740daa85189f2beb249ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:00.754691 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4395759135b84b5faf476339ce2cf6c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:00.755203 2667 ts_tablet_manager.cc:1918] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:56:00.755303 2667 log.cc:1196] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/605c555c03424367a5b719bde1b1c712
I20260926 20:56:00.755553 2667 ts_tablet_manager.cc:1939] T 605c555c03424367a5b719bde1b1c712 P 894967ab7a5b4f09b7309558d7ce327f: Deleting consensus metadata
I20260926 20:56:00.755940 2626 tablet_bootstrap.cc:492] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f: Bootstrap starting.
I20260926 20:56:00.755964 2627 tablet_bootstrap.cc:492] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47: Bootstrap starting.
I20260926 20:56:00.756332 2626 tablet_bootstrap.cc:654] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:00.756654 2104 catalog_manager.cc:4841] TS a4d8becac609479ba233629c5d5b8b47 (127.0.98.1:44501): tablet 605c555c03424367a5b719bde1b1c712 (table test-table [id=54b91ad832944fab801c1b39f36f2125]) successfully deleted
I20260926 20:56:00.756748 2104 catalog_manager.cc:4841] TS 894967ab7a5b4f09b7309558d7ce327f (127.0.98.3:40625): tablet 605c555c03424367a5b719bde1b1c712 (table test-table [id=54b91ad832944fab801c1b39f36f2125]) successfully deleted
I20260926 20:56:00.756773 2396 tablet_service.cc:1467] Processing CreateTablet for tablet 4395759135b84b5faf476339ce2cf6c1 (DEFAULT_TABLE table=test-table [id=1d6300439bc740daa85189f2beb249ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:00.756386 2627 tablet_bootstrap.cc:654] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:00.756887 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4395759135b84b5faf476339ce2cf6c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:00.757157 2665 tablet_replica.cc:330] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a: stopping tablet replica
I20260926 20:56:00.757277 2665 raft_consensus.cc:2229] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:00.757334 2665 raft_consensus.cc:2258] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:00.757337 2626 tablet_bootstrap.cc:492] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f: No bootstrap required, opened a new log
I20260926 20:56:00.757376 2626 ts_tablet_manager.cc:1397] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:00.757508 2626 raft_consensus.cc:348] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:00.757589 2626 raft_consensus.cc:374] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:00.757615 2626 raft_consensus.cc:729] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894967ab7a5b4f09b7309558d7ce327f, State: Initialized, Role: FOLLOWER
I20260926 20:56:00.757663 2626 consensus_queue.cc:260] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:00.757774 2626 ts_tablet_manager.cc:1428] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:00.758296 2625 tablet_bootstrap.cc:492] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a: Bootstrap starting.
I20260926 20:56:00.758381 2665 ts_tablet_manager.cc:1905] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:00.758723 2625 tablet_bootstrap.cc:654] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:00.759286 2625 tablet_bootstrap.cc:492] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a: No bootstrap required, opened a new log
I20260926 20:56:00.759339 2625 ts_tablet_manager.cc:1397] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:00.759492 2625 raft_consensus.cc:348] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:00.759585 2625 raft_consensus.cc:374] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:00.759610 2625 raft_consensus.cc:729] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d72172b78b754368ba4a8f3e3f44889a, State: Initialized, Role: FOLLOWER
I20260926 20:56:00.759632 2627 tablet_bootstrap.cc:492] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47: No bootstrap required, opened a new log
I20260926 20:56:00.759673 2627 ts_tablet_manager.cc:1397] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 20:56:00.759653 2625 consensus_queue.cc:260] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:00.759747 2625 ts_tablet_manager.cc:1428] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:00.759827 2665 ts_tablet_manager.cc:1918] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:56:00.759819 2627 raft_consensus.cc:348] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:00.759886 2627 raft_consensus.cc:374] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:00.759887 2665 log.cc:1196] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/605c555c03424367a5b719bde1b1c712
I20260926 20:56:00.759909 2627 raft_consensus.cc:729] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d8becac609479ba233629c5d5b8b47, State: Initialized, Role: FOLLOWER
I20260926 20:56:00.759960 2627 consensus_queue.cc:260] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:00.760062 2627 ts_tablet_manager.cc:1428] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:00.760139 2665 ts_tablet_manager.cc:1939] T 605c555c03424367a5b719bde1b1c712 P d72172b78b754368ba4a8f3e3f44889a: Deleting consensus metadata
I20260926 20:56:00.760591 2103 catalog_manager.cc:4841] TS d72172b78b754368ba4a8f3e3f44889a (127.0.98.2:45267): tablet 605c555c03424367a5b719bde1b1c712 (table test-table [id=54b91ad832944fab801c1b39f36f2125]) successfully deleted
I20260926 20:56:00.776579 2633 raft_consensus.cc:482] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:56:00.776705 2633 raft_consensus.cc:504] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:00.776875 2633 leader_election.cc:288] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4d8becac609479ba233629c5d5b8b47 (127.0.98.1:44501), d72172b78b754368ba4a8f3e3f44889a (127.0.98.2:45267)
I20260926 20:56:00.777118 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4395759135b84b5faf476339ce2cf6c1" candidate_uuid: "894967ab7a5b4f09b7309558d7ce327f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d72172b78b754368ba4a8f3e3f44889a" is_pre_election: true
I20260926 20:56:00.777127 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4395759135b84b5faf476339ce2cf6c1" candidate_uuid: "894967ab7a5b4f09b7309558d7ce327f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4d8becac609479ba233629c5d5b8b47" is_pre_election: true
I20260926 20:56:00.777220 2416 raft_consensus.cc:2454] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 894967ab7a5b4f09b7309558d7ce327f in term 0.
I20260926 20:56:00.777246 2275 raft_consensus.cc:2454] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 894967ab7a5b4f09b7309558d7ce327f in term 0.
I20260926 20:56:00.777369 2487 leader_election.cc:302] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 894967ab7a5b4f09b7309558d7ce327f, d72172b78b754368ba4a8f3e3f44889a; no voters:
I20260926 20:56:00.777599 2633 raft_consensus.cc:2781] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:56:00.777658 2633 raft_consensus.cc:482] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:56:00.777683 2633 raft_consensus.cc:3037] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:00.778537 2633 raft_consensus.cc:504] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:00.778700 2633 leader_election.cc:288] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [CANDIDATE]: Term 1 election: Requested vote from peers a4d8becac609479ba233629c5d5b8b47 (127.0.98.1:44501), d72172b78b754368ba4a8f3e3f44889a (127.0.98.2:45267)
I20260926 20:56:00.778887 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4395759135b84b5faf476339ce2cf6c1" candidate_uuid: "894967ab7a5b4f09b7309558d7ce327f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d72172b78b754368ba4a8f3e3f44889a"
I20260926 20:56:00.778899 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4395759135b84b5faf476339ce2cf6c1" candidate_uuid: "894967ab7a5b4f09b7309558d7ce327f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4d8becac609479ba233629c5d5b8b47"
I20260926 20:56:00.778972 2416 raft_consensus.cc:3037] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:00.779008 2275 raft_consensus.cc:3037] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:00.779685 2275 raft_consensus.cc:2454] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 894967ab7a5b4f09b7309558d7ce327f in term 1.
I20260926 20:56:00.779759 2416 raft_consensus.cc:2454] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 894967ab7a5b4f09b7309558d7ce327f in term 1.
I20260926 20:56:00.779892 2488 leader_election.cc:302] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 894967ab7a5b4f09b7309558d7ce327f, a4d8becac609479ba233629c5d5b8b47; no voters:
I20260926 20:56:00.779995 2633 raft_consensus.cc:2781] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:00.780066 2633 raft_consensus.cc:686] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [term 1 LEADER]: Becoming Leader. State: Replica: 894967ab7a5b4f09b7309558d7ce327f, State: Running, Role: LEADER
I20260926 20:56:00.780112 2633 consensus_queue.cc:237] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:00.780793 2118 catalog_manager.cc:5495] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f reported cstate change: term changed from 0 to 1, leader changed from <none> to 894967ab7a5b4f09b7309558d7ce327f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "894967ab7a5b4f09b7309558d7ce327f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } health_report { overall_health: HEALTHY } } }
I20260926 20:56:00.789816 2275 raft_consensus.cc:1261] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47 [term 1 FOLLOWER]: Refusing update from remote peer 894967ab7a5b4f09b7309558d7ce327f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:00.789806 2416 raft_consensus.cc:1261] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a [term 1 FOLLOWER]: Refusing update from remote peer 894967ab7a5b4f09b7309558d7ce327f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:00.790087 2663 consensus_queue.cc:1035] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:00.790212 2633 consensus_queue.cc:1035] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:00.792414 2651 mvcc.cc:205] Tried to move back new op lower bound from 7333708434593845248 to 7333708434555764736. Current Snapshot: MvccSnapshot[applied={T|T < 7333708434593845248}]
W20260926 20:56:00.795032 2650 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 20:56:02.313896 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:02.313998 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:02.314963 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c: Sending DeleteTablet for 3 replicas of tablet 4395759135b84b5faf476339ce2cf6c1
I20260926 20:56:02.315310 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 4395759135b84b5faf476339ce2cf6c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57774
I20260926 20:56:02.315385 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet 4395759135b84b5faf476339ce2cf6c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49002
I20260926 20:56:02.315393 2536 tablet_service.cc:1514] Processing DeleteTablet for tablet 4395759135b84b5faf476339ce2cf6c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45892
I20260926 20:56:02.315730 2696 tablet_replica.cc:330] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f: stopping tablet replica
I20260926 20:56:02.315515 2694 tablet_replica.cc:330] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47: stopping tablet replica
I20260926 20:56:02.315838 2694 raft_consensus.cc:2229] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:02.315884 2694 raft_consensus.cc:2258] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:02.315969 2696 raft_consensus.cc:2229] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:56:02.316092 2696 raft_consensus.cc:2258] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:02.316119 2694 ts_tablet_manager.cc:1905] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:02.316473 2696 ts_tablet_manager.cc:1905] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:02.317950 2694 ts_tablet_manager.cc:1918] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:56:02.318033 2694 log.cc:1196] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/wal/wals/4395759135b84b5faf476339ce2cf6c1
I20260926 20:56:02.318264 2694 ts_tablet_manager.cc:1939] T 4395759135b84b5faf476339ce2cf6c1 P a4d8becac609479ba233629c5d5b8b47: Deleting consensus metadata
I20260926 20:56:02.318647 2104 catalog_manager.cc:4841] TS a4d8becac609479ba233629c5d5b8b47 (127.0.98.1:44501): tablet 4395759135b84b5faf476339ce2cf6c1 (table test-table [id=1d6300439bc740daa85189f2beb249ed]) successfully deleted
I20260926 20:56:02.318882 2696 ts_tablet_manager.cc:1918] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:56:02.318961 2696 log.cc:1196] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/4395759135b84b5faf476339ce2cf6c1
I20260926 20:56:02.319221 2696 ts_tablet_manager.cc:1939] T 4395759135b84b5faf476339ce2cf6c1 P 894967ab7a5b4f09b7309558d7ce327f: Deleting consensus metadata
I20260926 20:56:02.319494 2695 tablet_replica.cc:330] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a: stopping tablet replica
I20260926 20:56:02.319576 2695 raft_consensus.cc:2229] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:02.319582 2104 catalog_manager.cc:4841] TS 894967ab7a5b4f09b7309558d7ce327f (127.0.98.3:40625): tablet 4395759135b84b5faf476339ce2cf6c1 (table test-table [id=1d6300439bc740daa85189f2beb249ed]) successfully deleted
I20260926 20:56:02.319635 2695 raft_consensus.cc:2258] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:02.319912 2695 ts_tablet_manager.cc:1905] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:02.321617 2695 ts_tablet_manager.cc:1918] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:56:02.321672 2695 log.cc:1196] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/4395759135b84b5faf476339ce2cf6c1
I20260926 20:56:02.321902 2695 ts_tablet_manager.cc:1939] T 4395759135b84b5faf476339ce2cf6c1 P d72172b78b754368ba4a8f3e3f44889a: Deleting consensus metadata
I20260926 20:56:02.322327 2103 catalog_manager.cc:4841] TS d72172b78b754368ba4a8f3e3f44889a (127.0.98.2:45267): tablet 4395759135b84b5faf476339ce2cf6c1 (table test-table [id=1d6300439bc740daa85189f2beb249ed]) successfully deleted
Sep 26 20:56:02 dist-test-slave-fl7g krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456162, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:56:02 dist-test-slave-fl7g krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456162, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:02.340803 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.333125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49436 (local address 127.0.98.62:38333)
0926 20:56:02.333268 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:02.333270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:02.333297 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:56:02.333372 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:02.333372 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:02.333384 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:02.333437 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:02.334011 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.334522 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.335340 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.335534 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.335588 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.338049 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:02.338051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:02.338051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:02.338058 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:02.339553 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:02.339911 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:02.339912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:02.339912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:02.339974 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:02.340198 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:02.340199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:02.340199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:02.340309 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:02.340385 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:02.340603 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 20:56:02.340680 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260926 20:56:02.342437 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:56:02.342584 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:56:02.343931 2256 tablet_service.cc:1467] Processing CreateTablet for tablet fcef7fa5a0344190ab0bc44e0b1d1d09 (DEFAULT_TABLE table=test-table [id=b9e394b4122942259fc61affeca70449]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:02.343987 2396 tablet_service.cc:1467] Processing CreateTablet for tablet fcef7fa5a0344190ab0bc44e0b1d1d09 (DEFAULT_TABLE table=test-table [id=b9e394b4122942259fc61affeca70449]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:02.343997 2536 tablet_service.cc:1467] Processing CreateTablet for tablet fcef7fa5a0344190ab0bc44e0b1d1d09 (DEFAULT_TABLE table=test-table [id=b9e394b4122942259fc61affeca70449]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:02.344079 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcef7fa5a0344190ab0bc44e0b1d1d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:02.344125 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcef7fa5a0344190ab0bc44e0b1d1d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:02.344125 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcef7fa5a0344190ab0bc44e0b1d1d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:02.345549 2707 tablet_bootstrap.cc:492] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f: Bootstrap starting.
I20260926 20:56:02.345687 2705 tablet_bootstrap.cc:492] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47: Bootstrap starting.
I20260926 20:56:02.345983 2707 tablet_bootstrap.cc:654] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:02.345992 2705 tablet_bootstrap.cc:654] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:02.346380 2706 tablet_bootstrap.cc:492] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a: Bootstrap starting.
I20260926 20:56:02.346594 2705 tablet_bootstrap.cc:492] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47: No bootstrap required, opened a new log
I20260926 20:56:02.346652 2705 ts_tablet_manager.cc:1397] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:02.346666 2707 tablet_bootstrap.cc:492] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f: No bootstrap required, opened a new log
I20260926 20:56:02.346715 2707 ts_tablet_manager.cc:1397] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:02.346772 2706 tablet_bootstrap.cc:654] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:02.346810 2705 raft_consensus.cc:348] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:02.346854 2707 raft_consensus.cc:348] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:02.346895 2705 raft_consensus.cc:374] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:02.346922 2705 raft_consensus.cc:729] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d8becac609479ba233629c5d5b8b47, State: Initialized, Role: FOLLOWER
I20260926 20:56:02.346925 2707 raft_consensus.cc:374] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:02.346946 2707 raft_consensus.cc:729] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894967ab7a5b4f09b7309558d7ce327f, State: Initialized, Role: FOLLOWER
I20260926 20:56:02.346993 2705 consensus_queue.cc:260] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:02.347013 2707 consensus_queue.cc:260] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:02.347170 2707 ts_tablet_manager.cc:1428] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:02.347169 2705 ts_tablet_manager.cc:1428] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:02.347674 2706 tablet_bootstrap.cc:492] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a: No bootstrap required, opened a new log
I20260926 20:56:02.347730 2706 ts_tablet_manager.cc:1397] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:02.347873 2706 raft_consensus.cc:348] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:02.347957 2706 raft_consensus.cc:374] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:02.347990 2706 raft_consensus.cc:729] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d72172b78b754368ba4a8f3e3f44889a, State: Initialized, Role: FOLLOWER
I20260926 20:56:02.348035 2706 consensus_queue.cc:260] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:02.348205 2706 ts_tablet_manager.cc:1428] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:02.419463 2712 raft_consensus.cc:482] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:56:02.419589 2712 raft_consensus.cc:504] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:02.419966 2712 leader_election.cc:288] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4d8becac609479ba233629c5d5b8b47 (127.0.98.1:44501), 894967ab7a5b4f09b7309558d7ce327f (127.0.98.3:40625)
I20260926 20:56:02.424624 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.420644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50951 (local address 127.0.98.3:40625)
0926 20:56:02.420833 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:02.420835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:02.420842 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:56:02.420865 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:02.420865 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:02.420873 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:02.420930 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:02.421137 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.421841 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.423586 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.424173 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.424251 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.424266 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:02.424488 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 20:56:02.424535 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":106,"threads_started":1}
I20260926 20:56:02.424634 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.420528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40625 (local address 127.0.98.2:50951)
0926 20:56:02.420690 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:02.420702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:02.420757 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:02.420955 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:02.420956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:02.420959 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:02.421039 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:02.421041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.421945 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.421946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:02.423476 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:02.423479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.424268 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.424269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:02.424352 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.424357 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:56:02.424450 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 20:56:02.424536 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260926 20:56:02.424778 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.420211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44501 (local address 127.0.98.2:51981)
0926 20:56:02.420527 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:02.420542 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:02.420620 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:02.420837 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:02.420838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:02.420841 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:02.420947 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:02.420949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.421945 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.421946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:02.423628 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:02.423630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.424460 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.424462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:02.424532 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.424536 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:56:02.424618 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:56:02.424668 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":56,"thread_start_us":143,"threads_started":1}
I20260926 20:56:02.425204 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.420316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51981 (local address 127.0.98.1:44501)
0926 20:56:02.420522 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:02.420524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:02.420621 (+ 97us) server_negotiation.cc:407] Connection header received
0926 20:56:02.420719 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:02.420720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:02.420733 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:02.420803 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:02.421015 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.421842 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.423712 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.424332 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.424413 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.424443 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:02.424849 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 20:56:02.424905 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":112,"threads_started":1}
I20260926 20:56:02.425398 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcef7fa5a0344190ab0bc44e0b1d1d09" candidate_uuid: "d72172b78b754368ba4a8f3e3f44889a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4d8becac609479ba233629c5d5b8b47" is_pre_election: true
I20260926 20:56:02.425519 2275 raft_consensus.cc:2454] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d72172b78b754368ba4a8f3e3f44889a in term 0.
I20260926 20:56:02.425463 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcef7fa5a0344190ab0bc44e0b1d1d09" candidate_uuid: "d72172b78b754368ba4a8f3e3f44889a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894967ab7a5b4f09b7309558d7ce327f" is_pre_election: true
I20260926 20:56:02.425606 2556 raft_consensus.cc:2454] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d72172b78b754368ba4a8f3e3f44889a in term 0.
I20260926 20:56:02.425724 2348 leader_election.cc:302] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4d8becac609479ba233629c5d5b8b47, d72172b78b754368ba4a8f3e3f44889a; no voters:
I20260926 20:56:02.425959 2712 raft_consensus.cc:2781] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:56:02.426011 2712 raft_consensus.cc:482] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:56:02.426041 2712 raft_consensus.cc:3037] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:02.426729 2712 raft_consensus.cc:504] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:02.426898 2712 leader_election.cc:288] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [CANDIDATE]: Term 1 election: Requested vote from peers a4d8becac609479ba233629c5d5b8b47 (127.0.98.1:44501), 894967ab7a5b4f09b7309558d7ce327f (127.0.98.3:40625)
I20260926 20:56:02.427069 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcef7fa5a0344190ab0bc44e0b1d1d09" candidate_uuid: "d72172b78b754368ba4a8f3e3f44889a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894967ab7a5b4f09b7309558d7ce327f"
I20260926 20:56:02.427069 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcef7fa5a0344190ab0bc44e0b1d1d09" candidate_uuid: "d72172b78b754368ba4a8f3e3f44889a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4d8becac609479ba233629c5d5b8b47"
I20260926 20:56:02.427147 2275 raft_consensus.cc:3037] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:02.427147 2556 raft_consensus.cc:3037] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:02.427907 2275 raft_consensus.cc:2454] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d72172b78b754368ba4a8f3e3f44889a in term 1.
I20260926 20:56:02.427946 2556 raft_consensus.cc:2454] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d72172b78b754368ba4a8f3e3f44889a in term 1.
I20260926 20:56:02.428115 2348 leader_election.cc:302] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4d8becac609479ba233629c5d5b8b47, d72172b78b754368ba4a8f3e3f44889a; no voters:
I20260926 20:56:02.428232 2712 raft_consensus.cc:2781] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:02.428460 2712 raft_consensus.cc:686] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [term 1 LEADER]: Becoming Leader. State: Replica: d72172b78b754368ba4a8f3e3f44889a, State: Running, Role: LEADER
I20260926 20:56:02.428563 2712 consensus_queue.cc:237] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:02.429346 2119 catalog_manager.cc:5495] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a reported cstate change: term changed from 0 to 1, leader changed from <none> to d72172b78b754368ba4a8f3e3f44889a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d72172b78b754368ba4a8f3e3f44889a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } health_report { overall_health: UNKNOWN } } }
I20260926 20:56:02.445001 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.442329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49008 (local address 127.0.98.2:45267)
0926 20:56:02.442487 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:02.442489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:02.442495 (+ 6us) server_negotiation.cc:407] Connection header received
0926 20:56:02.442529 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:02.442530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:02.442551 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:02.442672 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:02.442781 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.443281 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.444111 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.444323 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.444435 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.444643 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:02.444753 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:02.444843 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:02.444867 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 20:56:02.444889 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260926 20:56:02.450073 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.447732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50679 (local address 127.0.98.3:40625)
0926 20:56:02.447823 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:02.447825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:02.447895 (+ 70us) server_negotiation.cc:407] Connection header received
0926 20:56:02.447923 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:02.447924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:02.447934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:02.448000 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:02.448172 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.448660 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.449401 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.449601 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.449659 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.449845 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:02.449879 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:02.449937 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:02.449986 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 20:56:02.450010 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:56:02.450106 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.447665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40625 (local address 127.0.98.2:50679)
0926 20:56:02.447828 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:02.447838 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:02.447888 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:02.448010 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:02.448011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:02.448012 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:56:02.448084 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:02.448085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.448733 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.448734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:02.449316 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:02.449318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.449672 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.449673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:02.449722 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.449735 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:56:02.449944 (+ 209us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:56:02.449946 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:56:02.449979 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 20:56:02.450016 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260926 20:56:02.452468 2275 raft_consensus.cc:1261] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47 [term 1 FOLLOWER]: Refusing update from remote peer d72172b78b754368ba4a8f3e3f44889a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:02.452608 2556 raft_consensus.cc:1261] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f [term 1 FOLLOWER]: Refusing update from remote peer d72172b78b754368ba4a8f3e3f44889a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:02.452782 2712 consensus_queue.cc:1035] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:02.452932 2717 consensus_queue.cc:1035] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [LEADER]: Connected to new peer: Peer: permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:02.460152 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.457790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45267 (local address 127.0.98.3:46665)
0926 20:56:02.457917 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:02.457928 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:02.457979 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:02.458130 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:02.458131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:02.458132 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:56:02.458175 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:02.458176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.458786 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.458786 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:02.459337 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:02.459338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.459768 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.459769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:02.459833 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.459838 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:56:02.459998 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:56:02.460000 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:56:02.460045 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 20:56:02.460092 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260926 20:56:02.460150 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.457840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46665 (local address 127.0.98.2:45267)
0926 20:56:02.457882 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:02.457884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:02.458042 (+ 158us) server_negotiation.cc:407] Connection header received
0926 20:56:02.458068 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:02.458069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:02.458078 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:02.458124 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:02.458217 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.458721 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.459430 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.459694 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.459765 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.459913 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:02.459947 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:02.459998 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:02.460049 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 20:56:02.460071 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:56:02.466753 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:02.466872 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:02.467530 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c: Sending DeleteTablet for 3 replicas of tablet fcef7fa5a0344190ab0bc44e0b1d1d09
I20260926 20:56:02.467785 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet fcef7fa5a0344190ab0bc44e0b1d1d09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49002
I20260926 20:56:02.467830 2536 tablet_service.cc:1514] Processing DeleteTablet for tablet fcef7fa5a0344190ab0bc44e0b1d1d09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45892
I20260926 20:56:02.467820 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet fcef7fa5a0344190ab0bc44e0b1d1d09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57774
I20260926 20:56:02.467878 2695 tablet_replica.cc:330] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a: stopping tablet replica
I20260926 20:56:02.467921 2694 tablet_replica.cc:330] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47: stopping tablet replica
I20260926 20:56:02.467926 2695 raft_consensus.cc:2229] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:56:02.467964 2694 raft_consensus.cc:2229] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:02.468003 2695 raft_consensus.cc:2258] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:02.468003 2694 raft_consensus.cc:2258] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:02.468093 2696 tablet_replica.cc:330] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f: stopping tablet replica
I20260926 20:56:02.468148 2696 raft_consensus.cc:2229] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:02.468182 2696 raft_consensus.cc:2258] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:02.468375 2694 ts_tablet_manager.cc:1905] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:02.468470 2696 ts_tablet_manager.cc:1905] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:02.468467 2695 ts_tablet_manager.cc:1905] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:02.470429 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:56:02.470610 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:56:02.471997 2256 tablet_service.cc:1467] Processing CreateTablet for tablet a103203575a642b4abd0932534e9084c (DEFAULT_TABLE table=test-table [id=5372e95fea4f47a8860e18ee9b33b269]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:02.472008 2396 tablet_service.cc:1467] Processing CreateTablet for tablet a103203575a642b4abd0932534e9084c (DEFAULT_TABLE table=test-table [id=5372e95fea4f47a8860e18ee9b33b269]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:02.472019 2536 tablet_service.cc:1467] Processing CreateTablet for tablet a103203575a642b4abd0932534e9084c (DEFAULT_TABLE table=test-table [id=5372e95fea4f47a8860e18ee9b33b269]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:02.472477 2696 ts_tablet_manager.cc:1918] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:56:02.472476 2695 ts_tablet_manager.cc:1918] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:56:02.472478 2694 ts_tablet_manager.cc:1918] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:56:02.472535 2695 log.cc:1196] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/fcef7fa5a0344190ab0bc44e0b1d1d09
I20260926 20:56:02.472535 2696 log.cc:1196] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/fcef7fa5a0344190ab0bc44e0b1d1d09
I20260926 20:56:02.472540 2694 log.cc:1196] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-0/wal/wals/fcef7fa5a0344190ab0bc44e0b1d1d09
I20260926 20:56:02.472782 2696 ts_tablet_manager.cc:1939] T fcef7fa5a0344190ab0bc44e0b1d1d09 P 894967ab7a5b4f09b7309558d7ce327f: Deleting consensus metadata
I20260926 20:56:02.472782 2695 ts_tablet_manager.cc:1939] T fcef7fa5a0344190ab0bc44e0b1d1d09 P d72172b78b754368ba4a8f3e3f44889a: Deleting consensus metadata
I20260926 20:56:02.472784 2694 ts_tablet_manager.cc:1939] T fcef7fa5a0344190ab0bc44e0b1d1d09 P a4d8becac609479ba233629c5d5b8b47: Deleting consensus metadata
I20260926 20:56:02.472929 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a103203575a642b4abd0932534e9084c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:02.472942 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a103203575a642b4abd0932534e9084c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:02.474176 2706 tablet_bootstrap.cc:492] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a: Bootstrap starting.
I20260926 20:56:02.474197 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a103203575a642b4abd0932534e9084c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:02.474514 2104 catalog_manager.cc:4841] TS a4d8becac609479ba233629c5d5b8b47 (127.0.98.1:44501): tablet fcef7fa5a0344190ab0bc44e0b1d1d09 (table test-table [id=b9e394b4122942259fc61affeca70449]) successfully deleted
I20260926 20:56:02.474603 2706 tablet_bootstrap.cc:654] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:02.474933 2705 tablet_bootstrap.cc:492] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47: Bootstrap starting.
I20260926 20:56:02.475414 2707 tablet_bootstrap.cc:492] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f: Bootstrap starting.
I20260926 20:56:02.475745 2707 tablet_bootstrap.cc:654] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:02.475785 2705 tablet_bootstrap.cc:654] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:02.476233 2104 catalog_manager.cc:4841] TS 894967ab7a5b4f09b7309558d7ce327f (127.0.98.3:40625): tablet fcef7fa5a0344190ab0bc44e0b1d1d09 (table test-table [id=b9e394b4122942259fc61affeca70449]) successfully deleted
I20260926 20:56:02.476311 2706 tablet_bootstrap.cc:492] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a: No bootstrap required, opened a new log
I20260926 20:56:02.476372 2706 ts_tablet_manager.cc:1397] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:56:02.476568 2103 catalog_manager.cc:4841] TS d72172b78b754368ba4a8f3e3f44889a (127.0.98.2:45267): tablet fcef7fa5a0344190ab0bc44e0b1d1d09 (table test-table [id=b9e394b4122942259fc61affeca70449]) successfully deleted
I20260926 20:56:02.476577 2706 raft_consensus.cc:348] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:02.476661 2706 raft_consensus.cc:374] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:02.476688 2706 raft_consensus.cc:729] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d72172b78b754368ba4a8f3e3f44889a, State: Initialized, Role: FOLLOWER
I20260926 20:56:02.476730 2706 consensus_queue.cc:260] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:02.476832 2707 tablet_bootstrap.cc:492] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f: No bootstrap required, opened a new log
I20260926 20:56:02.476836 2706 ts_tablet_manager.cc:1428] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:02.476866 2707 ts_tablet_manager.cc:1397] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:02.476944 2707 raft_consensus.cc:348] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:02.476987 2707 raft_consensus.cc:374] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:02.477000 2707 raft_consensus.cc:729] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894967ab7a5b4f09b7309558d7ce327f, State: Initialized, Role: FOLLOWER
I20260926 20:56:02.477075 2707 consensus_queue.cc:260] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:02.477164 2707 ts_tablet_manager.cc:1428] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:02.477501 2705 tablet_bootstrap.cc:492] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47: No bootstrap required, opened a new log
I20260926 20:56:02.477560 2705 ts_tablet_manager.cc:1397] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 20:56:02.477718 2705 raft_consensus.cc:348] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:02.477790 2705 raft_consensus.cc:374] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:02.477811 2705 raft_consensus.cc:729] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d8becac609479ba233629c5d5b8b47, State: Initialized, Role: FOLLOWER
I20260926 20:56:02.477859 2705 consensus_queue.cc:260] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:02.477955 2705 ts_tablet_manager.cc:1428] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:02.559268 2711 raft_consensus.cc:482] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:56:02.559387 2711 raft_consensus.cc:504] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:02.559820 2711 leader_election.cc:288] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d72172b78b754368ba4a8f3e3f44889a (127.0.98.2:45267), 894967ab7a5b4f09b7309558d7ce327f (127.0.98.3:40625)
I20260926 20:56:02.564476 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.560115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40625 (local address 127.0.98.1:41307)
0926 20:56:02.560263 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:02.560277 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:02.560341 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:02.560630 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:02.560630 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:02.560632 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:02.560728 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:02.560729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.562094 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.562095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:02.563378 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:02.563381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.564169 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.564170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:02.564244 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.564248 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:56:02.564297 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 20:56:02.564337 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":19,"thread_start_us":69,"threads_started":1}
I20260926 20:56:02.564472 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.560270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41307 (local address 127.0.98.3:40625)
0926 20:56:02.560320 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:02.560322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:02.560335 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:56:02.560473 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:02.560475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:02.560485 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:02.560533 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:02.560813 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.561961 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.563505 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.564034 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.564083 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.564108 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:02.564323 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 20:56:02.564371 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:56:02.564781 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a103203575a642b4abd0932534e9084c" candidate_uuid: "a4d8becac609479ba233629c5d5b8b47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894967ab7a5b4f09b7309558d7ce327f" is_pre_election: true
I20260926 20:56:02.564893 2556 raft_consensus.cc:2454] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4d8becac609479ba233629c5d5b8b47 in term 0.
I20260926 20:56:02.565088 2208 leader_election.cc:302] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 894967ab7a5b4f09b7309558d7ce327f, a4d8becac609479ba233629c5d5b8b47; no voters:
I20260926 20:56:02.565227 2711 raft_consensus.cc:2781] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:56:02.565255 2711 raft_consensus.cc:482] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:56:02.565279 2711 raft_consensus.cc:3037] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:02.566148 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.560369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37907 (local address 127.0.98.2:45267)
0926 20:56:02.560429 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:02.560431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:02.560438 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:56:02.560472 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:02.560473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:02.560485 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:02.560574 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:02.562092 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.562893 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.565029 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.565700 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.565839 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.565852 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:02.566040 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 20:56:02.566084 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:56:02.566164 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.560115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45267 (local address 127.0.98.1:37907)
0926 20:56:02.560322 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:02.560332 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:02.560392 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:02.560623 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:02.560624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:02.560627 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:02.560727 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:02.560729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.562999 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.563000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:02.564950 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:02.564953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.565878 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.565879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:02.565946 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.565949 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:56:02.566033 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:56:02.566080 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":18,"thread_start_us":73,"threads_started":1}
I20260926 20:56:02.566692 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a103203575a642b4abd0932534e9084c" candidate_uuid: "a4d8becac609479ba233629c5d5b8b47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d72172b78b754368ba4a8f3e3f44889a" is_pre_election: true
I20260926 20:56:02.566805 2416 raft_consensus.cc:2454] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4d8becac609479ba233629c5d5b8b47 in term 0.
I20260926 20:56:02.569849 2711 raft_consensus.cc:504] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:02.570029 2711 leader_election.cc:288] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [CANDIDATE]: Term 1 election: Requested vote from peers d72172b78b754368ba4a8f3e3f44889a (127.0.98.2:45267), 894967ab7a5b4f09b7309558d7ce327f (127.0.98.3:40625)
I20260926 20:56:02.570343 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a103203575a642b4abd0932534e9084c" candidate_uuid: "a4d8becac609479ba233629c5d5b8b47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894967ab7a5b4f09b7309558d7ce327f"
I20260926 20:56:02.570474 2556 raft_consensus.cc:3037] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:02.571434 2556 raft_consensus.cc:2454] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4d8becac609479ba233629c5d5b8b47 in term 1.
I20260926 20:56:02.571674 2208 leader_election.cc:302] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 894967ab7a5b4f09b7309558d7ce327f, a4d8becac609479ba233629c5d5b8b47; no voters:
I20260926 20:56:02.571802 2711 raft_consensus.cc:2781] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:02.571990 2711 raft_consensus.cc:686] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [term 1 LEADER]: Becoming Leader. State: Replica: a4d8becac609479ba233629c5d5b8b47, State: Running, Role: LEADER
I20260926 20:56:02.572058 2711 consensus_queue.cc:237] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } }
I20260926 20:56:02.572958 2119 catalog_manager.cc:5495] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4d8becac609479ba233629c5d5b8b47 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a4d8becac609479ba233629c5d5b8b47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d8becac609479ba233629c5d5b8b47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 } health_report { overall_health: UNKNOWN } } }
I20260926 20:56:02.575194 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a103203575a642b4abd0932534e9084c" candidate_uuid: "a4d8becac609479ba233629c5d5b8b47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d72172b78b754368ba4a8f3e3f44889a"
I20260926 20:56:02.575277 2416 raft_consensus.cc:3037] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:02.599607 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.597125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57780 (local address 127.0.98.1:44501)
0926 20:56:02.597175 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:02.597177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:02.597185 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:56:02.597254 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:02.597255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:02.597275 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:02.597338 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:02.597423 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.597905 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.598697 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.598934 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.599013 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.599195 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:02.599306 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:02.599370 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:02.599520 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 20:56:02.599542 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:56:02.604961 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.602446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41177 (local address 127.0.98.3:40625)
0926 20:56:02.602510 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:02.602513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:02.602603 (+ 90us) server_negotiation.cc:407] Connection header received
0926 20:56:02.602681 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:02.602682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:02.602692 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:02.602769 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:02.602919 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.603391 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.604168 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.604362 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.604478 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.604674 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:02.604716 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:02.604786 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:02.604846 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 20:56:02.604869 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:56:02.604978 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.602265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40625 (local address 127.0.98.1:41177)
0926 20:56:02.602527 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:02.602539 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:02.602619 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:02.602772 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:02.602773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:02.602774 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:56:02.602843 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:02.602844 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.603481 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.603482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:02.604065 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:02.604066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.604508 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.604509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:02.604577 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.604590 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:56:02.604786 (+ 196us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:56:02.604788 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:56:02.604841 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 20:56:02.604882 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":100,"threads_started":1}
I20260926 20:56:02.607257 2556 raft_consensus.cc:1261] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f [term 1 FOLLOWER]: Refusing update from remote peer a4d8becac609479ba233629c5d5b8b47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:02.607556 2735 consensus_queue.cc:1035] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "894967ab7a5b4f09b7309558d7ce327f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:02.614854 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.612452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58933 (local address 127.0.98.1:44501)
0926 20:56:02.612493 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:02.612495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:02.612505 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:56:02.612567 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:02.612568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:02.612579 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:02.612646 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:02.612756 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.613232 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.614080 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.614398 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.614462 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.614628 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:02.614669 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:02.614726 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:02.614776 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 20:56:02.614800 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:56:02.614868 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:02.612357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44501 (local address 127.0.98.3:58933)
0926 20:56:02.612459 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:02.612471 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:02.612508 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:02.612651 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:02.612652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:02.612653 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:56:02.612701 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:02.612702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.613318 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.613319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:02.613981 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:02.613984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:02.614505 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:02.614506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:02.614568 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:02.614574 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:56:02.614737 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:56:02.614738 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:56:02.614773 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 20:56:02.614812 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
W20260926 20:56:02.615406 2680 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 20:56:02.666893 2416 raft_consensus.cc:2454] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4d8becac609479ba233629c5d5b8b47 in term 1.
I20260926 20:56:02.667228 2415 raft_consensus.cc:1261] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a [term 1 FOLLOWER]: Refusing update from remote peer a4d8becac609479ba233629c5d5b8b47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:02.667802 2735 consensus_queue.cc:1035] T a103203575a642b4abd0932534e9084c P a4d8becac609479ba233629c5d5b8b47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d72172b78b754368ba4a8f3e3f44889a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:04.139546 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:04.139653 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:04.140704 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c3ff07ee5164e36a26708f88f09ae0c: Sending DeleteTablet for 3 replicas of tablet a103203575a642b4abd0932534e9084c
I20260926 20:56:04.140959 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet a103203575a642b4abd0932534e9084c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57774
I20260926 20:56:04.141000 2536 tablet_service.cc:1514] Processing DeleteTablet for tablet a103203575a642b4abd0932534e9084c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45892
I20260926 20:56:04.141100 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet a103203575a642b4abd0932534e9084c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49002
I20260926 20:56:04.141199 2768 tablet_replica.cc:330] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f: stopping tablet replica
I20260926 20:56:04.141254 2768 raft_consensus.cc:2229] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:04.141299 2768 raft_consensus.cc:2258] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:04.141290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 2191
I20260926 20:56:04.141309 2769 tablet_replica.cc:330] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a: stopping tablet replica
I20260926 20:56:04.141364 2769 raft_consensus.cc:2229] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:04.141937 2769 raft_consensus.cc:2258] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:04.142205 2769 ts_tablet_manager.cc:1905] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:04.143235 2768 ts_tablet_manager.cc:1905] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:04.143860 2769 ts_tablet_manager.cc:1918] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:56:04.143992 2769 log.cc:1196] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/a103203575a642b4abd0932534e9084c
I20260926 20:56:04.144233 2769 ts_tablet_manager.cc:1939] T a103203575a642b4abd0932534e9084c P d72172b78b754368ba4a8f3e3f44889a: Deleting consensus metadata
I20260926 20:56:04.144622 2103 catalog_manager.cc:4841] TS d72172b78b754368ba4a8f3e3f44889a (127.0.98.2:45267): tablet a103203575a642b4abd0932534e9084c (table test-table [id=5372e95fea4f47a8860e18ee9b33b269]) successfully deleted
I20260926 20:56:04.145171 2768 ts_tablet_manager.cc:1918] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:56:04.145241 2768 log.cc:1196] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/a103203575a642b4abd0932534e9084c
I20260926 20:56:04.145470 2768 ts_tablet_manager.cc:1939] T a103203575a642b4abd0932534e9084c P 894967ab7a5b4f09b7309558d7ce327f: Deleting consensus metadata
I20260926 20:56:04.146188 2104 catalog_manager.cc:4841] TS 894967ab7a5b4f09b7309558d7ce327f (127.0.98.3:40625): tablet a103203575a642b4abd0932534e9084c (table test-table [id=5372e95fea4f47a8860e18ee9b33b269]) successfully deleted
W20260926 20:56:04.149298 2104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44501 (error 108)
I20260926 20:56:04.149545 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 2331
I20260926 20:56:04.151042 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:04.150739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44501 (local address 127.0.0.1:57790)
0926 20:56:04.150963 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:04.151005 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":72,"threads_started":1}
W20260926 20:56:04.151165 2104 catalog_manager.cc:4584] TS a4d8becac609479ba233629c5d5b8b47 (127.0.98.1:44501): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a103203575a642b4abd0932534e9084c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44501: connect: Connection refused (error 111)
I20260926 20:56:04.155313 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 2471
I20260926 20:56:04.161538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 2087
2026-09-26T20:56:04Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5590 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:56:04 dist-test-slave-fl7g krb5kdc[2774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:56:04 dist-test-slave-fl7g krb5kdc[2774](info): set up 2 sockets
Sep 26 20:56:04 dist-test-slave-fl7g krb5kdc[2774](info): commencing operation
krb5kdc: starting...
W20260926 20:56:05.204334 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:56:05 dist-test-slave-fl7g krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:56:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:56:05.331923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43281
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:05.414986 2787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:05.415275 2787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:05.415315 2787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:05.416697 2787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:05.416772 2787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:05.416795 2787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:05.416816 2787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:05.416847 2787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:05.418506 2787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:05.418826 2787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:05.419081 2787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:56:05.424422 2787 server_base.cc:1034] running on GCE node
W20260926 20:56:05.424382 2792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:05.424369 2795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:05.424382 2793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:05.424852 2787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:05.425195 2787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:05.426390 2787 hybrid_clock.cc:648] HybridClock initialized: now 1790456165426394 us; error 43 us; skew 500 ppm
Sep 26 20:56:05 dist-test-slave-fl7g krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:05.434605 2787 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:56:05.435221 2787 webserver.cc:460] Webserver started at http://127.0.98.62:39293/ using document root <none> and password file <none>
I20260926 20:56:05.435490 2787 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:05.435565 2787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:05.435726 2787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:05.436955 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "412b965331844bf9812cfbeb6fa513ab"
format_stamp: "Formatted at 2026-09-26 20:56:05 on dist-test-slave-fl7g"
server_key: "dad8f335bc264c39effa5d77642758b3"
server_key_iv: "5b11347503531842e4959fa186d99dd7"
server_key_version: "encryptionkey@0"
I20260926 20:56:05.437338 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "412b965331844bf9812cfbeb6fa513ab"
format_stamp: "Formatted at 2026-09-26 20:56:05 on dist-test-slave-fl7g"
server_key: "dad8f335bc264c39effa5d77642758b3"
server_key_iv: "5b11347503531842e4959fa186d99dd7"
server_key_version: "encryptionkey@0"
I20260926 20:56:05.438798 2787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:56:05.439644 2802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:05.439867 2787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:05.439980 2787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "412b965331844bf9812cfbeb6fa513ab"
format_stamp: "Formatted at 2026-09-26 20:56:05 on dist-test-slave-fl7g"
server_key: "dad8f335bc264c39effa5d77642758b3"
server_key_iv: "5b11347503531842e4959fa186d99dd7"
server_key_version: "encryptionkey@0"
I20260926 20:56:05.440062 2787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:05.449892 2787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:05.452488 2787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:05.452670 2787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:05.456907 2787 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43281
I20260926 20:56:05.457010 2854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43281 every 8 connection(s)
I20260926 20:56:05.457494 2787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:05.458184 2855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:05.460590 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab: Bootstrap starting.
I20260926 20:56:05.461212 2855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:05.461524 2855 log.cc:824] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:05.462312 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab: No bootstrap required, opened a new log
I20260926 20:56:05.464025 2855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412b965331844bf9812cfbeb6fa513ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } }
I20260926 20:56:05.464174 2855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:05.464196 2855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412b965331844bf9812cfbeb6fa513ab, State: Initialized, Role: FOLLOWER
I20260926 20:56:05.464376 2855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412b965331844bf9812cfbeb6fa513ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } }
I20260926 20:56:05.464522 2855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:05.464569 2855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:05.464622 2855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:05.465574 2855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412b965331844bf9812cfbeb6fa513ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } }
I20260926 20:56:05.465694 2855 leader_election.cc:302] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 412b965331844bf9812cfbeb6fa513ab; no voters:
I20260926 20:56:05.465896 2855 leader_election.cc:288] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:05.465950 2858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:05.466166 2858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab [term 1 LEADER]: Becoming Leader. State: Replica: 412b965331844bf9812cfbeb6fa513ab, State: Running, Role: LEADER
I20260926 20:56:05.466183 2855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:05.466284 2858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412b965331844bf9812cfbeb6fa513ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } }
I20260926 20:56:05.466714 2859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "412b965331844bf9812cfbeb6fa513ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412b965331844bf9812cfbeb6fa513ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } } }
I20260926 20:56:05.466826 2859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:05.466732 2860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 412b965331844bf9812cfbeb6fa513ab. Latest consensus state: current_term: 1 leader_uuid: "412b965331844bf9812cfbeb6fa513ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412b965331844bf9812cfbeb6fa513ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43281 } } }
I20260926 20:56:05.467053 2860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:05.467049 2867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:05.467453 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 2787
I20260926 20:56:05.467545 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:56:05.467630 2867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:05.467775 392 external_mini_cluster.cc:1428] Setting key f0f2d91f960c6613c5d0775d4e0d7299
I20260926 20:56:05.469352 2867 catalog_manager.cc:1347] Generated new cluster ID: 66796beead934c04a04bb7ec02711caa
I20260926 20:56:05.469405 2867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:56:05 dist-test-slave-fl7g krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:05.479298 2867 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:05.479889 2867 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:05.480515 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:05.468305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49766 (local address 127.0.98.62:43281)
0926 20:56:05.468680 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:05.468690 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:05.468725 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:56:05.468855 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:05.468861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:05.468915 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:05.469092 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:05.469931 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:05.471373 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:05.472524 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:05.473188 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:05.473298 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:05.476308 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:05.476314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:05.476315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:05.476332 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:05.478532 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:05.478984 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:05.478988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:05.478988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:05.479124 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:05.479350 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:05.479351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:05.479352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:05.479730 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:05.479892 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:05.480065 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 20:56:05.480232 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":74,"threads_started":1}
I20260926 20:56:05.488559 2867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 412b965331844bf9812cfbeb6fa513ab: Generated new TSK 0
I20260926 20:56:05.488809 2867 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:56:05.520542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43281
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:05.605183 2881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:05.605444 2881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:05.605489 2881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:05.606914 2881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:05.606988 2881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:05.607036 2881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:56:05.608681 2881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:05.608968 2881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:05.609220 2881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:05.614444 2886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:05.614531 2887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:05.614444 2889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:05.614645 2881 server_base.cc:1034] running on GCE node
I20260926 20:56:05.614944 2881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:05.615237 2881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:05.616483 2881 hybrid_clock.cc:648] HybridClock initialized: now 1790456165616455 us; error 51 us; skew 500 ppm
Sep 26 20:56:05 dist-test-slave-fl7g krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:05.619112 2881 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:56:05.619604 2881 webserver.cc:460] Webserver started at http://127.0.98.1:43097/ using document root <none> and password file <none>
I20260926 20:56:05.619851 2881 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:05.619896 2881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:05.620047 2881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:05.621258 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "1b25b30e59394bbcb99f8f27a3b32072"
format_stamp: "Formatted at 2026-09-26 20:56:05 on dist-test-slave-fl7g"
server_key: "8bc093db380c3fb1f8e58cb6d9d12c86"
server_key_iv: "bdf089eac112b95f392aa9c008bc323d"
server_key_version: "encryptionkey@0"
I20260926 20:56:05.621639 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1b25b30e59394bbcb99f8f27a3b32072"
format_stamp: "Formatted at 2026-09-26 20:56:05 on dist-test-slave-fl7g"
server_key: "8bc093db380c3fb1f8e58cb6d9d12c86"
server_key_iv: "bdf089eac112b95f392aa9c008bc323d"
server_key_version: "encryptionkey@0"
I20260926 20:56:05.623085 2881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:56:05.623917 2896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:05.624116 2881 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:05.624193 2881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "1b25b30e59394bbcb99f8f27a3b32072"
format_stamp: "Formatted at 2026-09-26 20:56:05 on dist-test-slave-fl7g"
server_key: "8bc093db380c3fb1f8e58cb6d9d12c86"
server_key_iv: "bdf089eac112b95f392aa9c008bc323d"
server_key_version: "encryptionkey@0"
I20260926 20:56:05.624291 2881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:05.635689 2881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:05.637820 2881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:05.637988 2881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:05.638233 2881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:05.638566 2881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:05.638623 2881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:05.638669 2881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:05.638696 2881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:05.644970 2881 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37909
I20260926 20:56:05.645015 3009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37909 every 8 connection(s)
I20260926 20:56:05.645351 2881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:56:05.645941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 2881
I20260926 20:56:05.646050 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:56:05.646270 392 external_mini_cluster.cc:1428] Setting key a1eab9f11226159bd2cfa69cf3fb06ac
Sep 26 20:56:05 dist-test-slave-fl7g krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 20:56:05.655807 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:05.646516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59567 (local address 127.0.98.62:43281)
0926 20:56:05.646568 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:05.646570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:05.646583 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:56:05.647600 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:05.647601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:05.647614 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:05.647665 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:05.648330 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:05.648896 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:05.649925 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:05.650136 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:05.650238 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:05.652703 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:05.652705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:05.652705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:05.652714 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:05.654264 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:05.654692 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:05.654693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:05.654694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:05.654780 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:05.655047 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:05.655048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:05.655048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:05.655176 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:05.655257 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:05.655542 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 20:56:05.655635 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 20:56:05.655939 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:05.646120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.1:59567)
0926 20:56:05.646434 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:05.646500 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:05.647468 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:05.647761 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:05.647765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:05.648023 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:05.648228 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:05.648232 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:05.648995 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:05.648996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:05.649823 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:05.649825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:05.650251 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:05.650252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:05.650374 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:05.650933 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:05.650942 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:05.652594 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:05.654351 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:05.654359 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:05.654361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:05.654594 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:05.654850 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:05.654851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:05.654851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:05.654975 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:05.655246 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:05.655253 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:05.655405 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 20:56:05.655520 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:56:05.655703 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 20:56:05.656561 3010 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43281
I20260926 20:56:05.656661 3010 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:05.656910 3010 heartbeater.cc:511] Master 127.0.98.62:43281 requested a full tablet report, sending...
I20260926 20:56:05.657505 2819 ts_manager.cc:194] Registered new tserver with Master: 1b25b30e59394bbcb99f8f27a3b32072 (127.0.98.1:37909)
I20260926 20:56:05.658136 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:56:05.672255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43281
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:05.755437 3015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:05.755668 3015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:05.755693 3015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:05.756981 3015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:05.757026 3015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:05.757054 3015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:56:05.758560 3015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:05.758764 3015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:05.758946 3015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:05.764071 3021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:05.764058 3023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:05.764075 3020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:05.764528 3015 server_base.cc:1034] running on GCE node
I20260926 20:56:05.764739 3015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:05.764959 3015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:05.766158 3015 hybrid_clock.cc:648] HybridClock initialized: now 1790456165766128 us; error 39 us; skew 500 ppm
Sep 26 20:56:05 dist-test-slave-fl7g krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:05.769062 3015 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:56:05.769625 3015 webserver.cc:460] Webserver started at http://127.0.98.2:42845/ using document root <none> and password file <none>
I20260926 20:56:05.769829 3015 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:05.769870 3015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:05.769963 3015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:05.770988 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "e41b784b2cb4435bbc7dcfbb119db8a1"
format_stamp: "Formatted at 2026-09-26 20:56:05 on dist-test-slave-fl7g"
server_key: "b06546b2225e01e6a6310e4f11d32315"
server_key_iv: "cdeeb61ab267d5bbaccd5dd28bde8208"
server_key_version: "encryptionkey@0"
I20260926 20:56:05.771304 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e41b784b2cb4435bbc7dcfbb119db8a1"
format_stamp: "Formatted at 2026-09-26 20:56:05 on dist-test-slave-fl7g"
server_key: "b06546b2225e01e6a6310e4f11d32315"
server_key_iv: "cdeeb61ab267d5bbaccd5dd28bde8208"
server_key_version: "encryptionkey@0"
I20260926 20:56:05.772606 3015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:56:05.773293 3030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:05.773499 3015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:05.773568 3015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "e41b784b2cb4435bbc7dcfbb119db8a1"
format_stamp: "Formatted at 2026-09-26 20:56:05 on dist-test-slave-fl7g"
server_key: "b06546b2225e01e6a6310e4f11d32315"
server_key_iv: "cdeeb61ab267d5bbaccd5dd28bde8208"
server_key_version: "encryptionkey@0"
I20260926 20:56:05.773659 3015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:05.786299 3015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:05.788640 3015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:05.788802 3015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:05.789057 3015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:05.789389 3015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:05.789424 3015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:05.789448 3015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:05.789495 3015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:05.795127 3015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43053
I20260926 20:56:05.795172 3143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43053 every 8 connection(s)
I20260926 20:56:05.795516 3015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:56:05.797413 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 3015
I20260926 20:56:05.797495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:56:05.797714 392 external_mini_cluster.cc:1428] Setting key 9a4f6c9808742bcc8c1b24653bf9093f
Sep 26 20:56:05 dist-test-slave-fl7g krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 20:56:05.806864 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:05.796303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33751 (local address 127.0.98.62:43281)
0926 20:56:05.796368 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:05.796370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:05.796613 (+ 243us) server_negotiation.cc:407] Connection header received
0926 20:56:05.797649 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:05.797651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:05.797663 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:05.797716 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:05.798504 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:05.799300 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:05.800885 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:05.801076 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:05.801139 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:05.803854 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:05.803855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:05.803856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:05.803864 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:05.805369 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:05.805808 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:05.805809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:05.805809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:05.805872 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:05.806138 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:05.806139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:05.806139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:05.806295 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:05.806356 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:05.806722 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 20:56:05.806794 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:56:05.807034 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:05.796275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.2:33751)
0926 20:56:05.796487 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:05.796525 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:05.797523 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:05.797808 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:05.797812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:05.798168 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:05.798398 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:05.798404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:05.799412 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:05.799413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:05.800763 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:05.800766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:05.801150 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:05.801152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:05.801315 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:05.802060 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:05.802071 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:05.803741 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:05.805474 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:05.805482 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:05.805485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:05.805728 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:05.805940 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:05.805941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:05.805942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:05.806068 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:05.806358 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:05.806367 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:05.806542 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 20:56:05.806670 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 20:56:05.806819 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":87,"threads_started":1}
I20260926 20:56:05.807612 3144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43281
I20260926 20:56:05.807737 3144 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:05.807929 3144 heartbeater.cc:511] Master 127.0.98.62:43281 requested a full tablet report, sending...
I20260926 20:56:05.808449 2819 ts_manager.cc:194] Registered new tserver with Master: e41b784b2cb4435bbc7dcfbb119db8a1 (127.0.98.2:43053)
I20260926 20:56:05.808837 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:56:05.827870 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43281
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:05.912151 3149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:05.912421 3149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:05.912483 3149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:05.913718 3149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:05.913808 3149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:05.913856 3149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:56:05.915493 3149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:05.915793 3149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:05.916059 3149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:05.921214 3157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:05.921252 3155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:05.921201 3154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:05.921456 3149 server_base.cc:1034] running on GCE node
I20260926 20:56:05.921706 3149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:05.921972 3149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:05.923146 3149 hybrid_clock.cc:648] HybridClock initialized: now 1790456165923127 us; error 44 us; skew 500 ppm
Sep 26 20:56:05 dist-test-slave-fl7g krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:05.925899 3149 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:56:05.926316 3149 webserver.cc:460] Webserver started at http://127.0.98.3:46599/ using document root <none> and password file <none>
I20260926 20:56:05.926563 3149 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:05.926610 3149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:05.926707 3149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:05.927829 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "0445fc11bc2f482988b7b6b04e39bfda"
format_stamp: "Formatted at 2026-09-26 20:56:05 on dist-test-slave-fl7g"
server_key: "299650e27077c1de1b123dd85084b655"
server_key_iv: "822d87849866a7624f92ef7406fae859"
server_key_version: "encryptionkey@0"
I20260926 20:56:05.928143 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0445fc11bc2f482988b7b6b04e39bfda"
format_stamp: "Formatted at 2026-09-26 20:56:05 on dist-test-slave-fl7g"
server_key: "299650e27077c1de1b123dd85084b655"
server_key_iv: "822d87849866a7624f92ef7406fae859"
server_key_version: "encryptionkey@0"
I20260926 20:56:05.929620 3149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:56:05.930473 3164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:05.930754 3149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:05.930853 3149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "0445fc11bc2f482988b7b6b04e39bfda"
format_stamp: "Formatted at 2026-09-26 20:56:05 on dist-test-slave-fl7g"
server_key: "299650e27077c1de1b123dd85084b655"
server_key_iv: "822d87849866a7624f92ef7406fae859"
server_key_version: "encryptionkey@0"
I20260926 20:56:05.930933 3149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:05.946483 3149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:05.949000 3149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:05.949159 3149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:05.949410 3149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:05.949734 3149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:05.949788 3149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:05.949831 3149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:05.949853 3149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:05.955736 3149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38315
I20260926 20:56:05.955780 3277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38315 every 8 connection(s)
I20260926 20:56:05.956107 3149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:56:05 dist-test-slave-fl7g krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:05.963096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 3149
I20260926 20:56:05.963217 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:56:05.963807 392 external_mini_cluster.cc:1428] Setting key 03bc7ac85a5debf4313817f27aae9c7f
I20260926 20:56:05.966198 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:05.957001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60813 (local address 127.0.98.62:43281)
0926 20:56:05.957062 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:05.957064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:05.957261 (+ 197us) server_negotiation.cc:407] Connection header received
0926 20:56:05.958247 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:05.958248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:05.958259 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:05.958334 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:05.958959 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:05.959461 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:05.960535 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:05.960731 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:05.960790 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:05.962964 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:05.962966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:05.962967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:05.962976 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:05.964636 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:05.965119 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:05.965120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:05.965120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:05.965187 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:05.965451 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:05.965452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:05.965452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:05.965596 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:05.965684 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:05.966013 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 20:56:05.966098 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:56:05.966445 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:05.956963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43281 (local address 127.0.98.3:60813)
0926 20:56:05.957125 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:05.957180 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:05.958139 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:05.958415 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:05.958418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:05.958687 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:05.958882 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:05.958886 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:05.959541 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:05.959542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:05.960380 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:05.960382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:05.960804 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:05.960805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:05.960911 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:05.961414 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:05.961426 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:05.962848 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:05.964758 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:05.964769 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:05.964777 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:05.965017 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:05.965255 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:05.965256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:05.965257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:05.965381 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:05.965677 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:05.965684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:05.965876 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 20:56:05.965993 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:56:05.966180 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260926 20:56:05.966985 3278 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43281
I20260926 20:56:05.967093 3278 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:05.967280 3278 heartbeater.cc:511] Master 127.0.98.62:43281 requested a full tablet report, sending...
I20260926 20:56:05.967645 2819 ts_manager.cc:194] Registered new tserver with Master: 0445fc11bc2f482988b7b6b04e39bfda (127.0.98.3:38315)
I20260926 20:56:05.968232 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60813
I20260926 20:56:05.975700 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:56:05.986976 2877 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:05.985906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49780 (local address 127.0.98.62:43281)
0926 20:56:05.985994 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:05.985997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:05.986113 (+ 116us) server_negotiation.cc:407] Connection header received
0926 20:56:05.986218 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:05.986219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:05.986229 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:05.986293 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:05.986888 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49780: BlockingRecv error: recv got EOF from 127.0.0.1:49780 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:56:05.987825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 2881
I20260926 20:56:05.992255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 3015
I20260926 20:56:05.996902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 3149
I20260926 20:56:06.001292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 2787
2026-09-26T20:56:06Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1838 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:56:06 dist-test-slave-fl7g krb5kdc[3290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:56:06 dist-test-slave-fl7g krb5kdc[3290](info): set up 2 sockets
Sep 26 20:56:06 dist-test-slave-fl7g krb5kdc[3290](info): commencing operation
krb5kdc: starting...
W20260926 20:56:07.044339 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:56:07 dist-test-slave-fl7g krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:56:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:56:07.172467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33531
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:07.256704 3303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:07.257052 3303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:07.257112 3303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:07.258473 3303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:07.258558 3303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:07.258586 3303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:07.258618 3303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:07.258641 3303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:07.260313 3303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:07.260645 3303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:07.260897 3303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:56:07.266266 3303 server_base.cc:1034] running on GCE node
W20260926 20:56:07.266225 3308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:07.266243 3309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:07.266243 3311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:07.266850 3303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:07.267310 3303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:07.268532 3303 hybrid_clock.cc:648] HybridClock initialized: now 1790456167268506 us; error 35 us; skew 500 ppm
Sep 26 20:56:07 dist-test-slave-fl7g krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:07.271484 3303 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:56:07.272064 3303 webserver.cc:460] Webserver started at http://127.0.98.62:45229/ using document root <none> and password file <none>
I20260926 20:56:07.272341 3303 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:07.272395 3303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:07.272627 3303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:07.273797 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "f6ed729329f14ac99ff6e2384b9c2d70"
format_stamp: "Formatted at 2026-09-26 20:56:07 on dist-test-slave-fl7g"
server_key: "adb6b941a8708b4eeea06e1141e08138"
server_key_iv: "6694014459c3787ac50be7ad2f052de0"
server_key_version: "encryptionkey@0"
I20260926 20:56:07.274214 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "f6ed729329f14ac99ff6e2384b9c2d70"
format_stamp: "Formatted at 2026-09-26 20:56:07 on dist-test-slave-fl7g"
server_key: "adb6b941a8708b4eeea06e1141e08138"
server_key_iv: "6694014459c3787ac50be7ad2f052de0"
server_key_version: "encryptionkey@0"
I20260926 20:56:07.275781 3303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:56:07.276772 3318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:07.276996 3303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:07.277117 3303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "f6ed729329f14ac99ff6e2384b9c2d70"
format_stamp: "Formatted at 2026-09-26 20:56:07 on dist-test-slave-fl7g"
server_key: "adb6b941a8708b4eeea06e1141e08138"
server_key_iv: "6694014459c3787ac50be7ad2f052de0"
server_key_version: "encryptionkey@0"
I20260926 20:56:07.277211 3303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:07.290063 3303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:07.292637 3303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:07.292757 3303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:07.296828 3303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33531
I20260926 20:56:07.296998 3370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33531 every 8 connection(s)
I20260926 20:56:07.297266 3303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:07.297619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 3303
I20260926 20:56:07.297712 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:56:07.297901 392 external_mini_cluster.cc:1428] Setting key 879c936b825aa164c48a443b6bcaab12
I20260926 20:56:07.298346 3371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:07.301776 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70: Bootstrap starting.
I20260926 20:56:07.302518 3371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:07.302821 3371 log.cc:824] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:07.303531 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70: No bootstrap required, opened a new log
Sep 26 20:56:07 dist-test-slave-fl7g krb5kdc[3290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:07.305877 3371 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ed729329f14ac99ff6e2384b9c2d70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33531 } }
I20260926 20:56:07.306053 3371 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:07.306108 3371 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ed729329f14ac99ff6e2384b9c2d70, State: Initialized, Role: FOLLOWER
I20260926 20:56:07.306234 3371 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ed729329f14ac99ff6e2384b9c2d70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33531 } }
I20260926 20:56:07.306320 3371 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:07.306360 3371 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:07.306414 3371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:07.307111 3371 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ed729329f14ac99ff6e2384b9c2d70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33531 } }
I20260926 20:56:07.307202 3371 leader_election.cc:302] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6ed729329f14ac99ff6e2384b9c2d70; no voters:
I20260926 20:56:07.307422 3371 leader_election.cc:288] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:07.307502 3376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:07.307678 3376 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70 [term 1 LEADER]: Becoming Leader. State: Replica: f6ed729329f14ac99ff6e2384b9c2d70, State: Running, Role: LEADER
I20260926 20:56:07.307724 3371 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:07.307901 3376 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ed729329f14ac99ff6e2384b9c2d70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33531 } }
I20260926 20:56:07.308358 3376 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6ed729329f14ac99ff6e2384b9c2d70. Latest consensus state: current_term: 1 leader_uuid: "f6ed729329f14ac99ff6e2384b9c2d70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ed729329f14ac99ff6e2384b9c2d70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33531 } } }
I20260926 20:56:07.308367 3378 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6ed729329f14ac99ff6e2384b9c2d70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ed729329f14ac99ff6e2384b9c2d70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33531 } } }
I20260926 20:56:07.308583 3378 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:07.308810 3376 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:07.308986 3385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:07.309566 3385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:07.310297 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:07.298457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47212 (local address 127.0.98.62:33531)
0926 20:56:07.298647 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:07.298658 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:07.298695 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:56:07.299140 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:07.299147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:07.299194 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:07.299311 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:07.300181 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:07.301303 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:07.302378 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:07.302895 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:07.303040 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:07.305793 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:07.305797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:07.305797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:07.305809 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:07.308043 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:07.308547 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:07.308550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:07.308551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:07.308680 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:07.308932 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:07.308933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:07.308933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:07.309261 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:07.309875 (+ 614us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:07.309948 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 20:56:07.310088 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260926 20:56:07.311304 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:56:07.311570 3385 catalog_manager.cc:1347] Generated new cluster ID: 5a92b2aecea346d0ba9711e1fa20ff0f
I20260926 20:56:07.311617 3385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:56:07.324023 3385 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:07.324731 3385 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:07.326048 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:07.321842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47226 (local address 127.0.98.62:33531)
0926 20:56:07.321892 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:07.321894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:07.322104 (+ 210us) server_negotiation.cc:407] Connection header received
0926 20:56:07.322195 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:07.322196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:07.322209 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:07.322261 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:07.322694 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:07.323215 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:07.324232 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:07.324628 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:07.324714 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:07.325521 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:07.325522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:07.325522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:07.325763 (+ 241us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:07.325783 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:07.325791 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:07.325858 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:07.325920 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 20:56:07.325969 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:56:07.330927 3385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6ed729329f14ac99ff6e2384b9c2d70: Generated new TSK 0
I20260926 20:56:07.331188 3385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:56:07.334605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 3303
2026-09-26T20:56:07Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1331 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:56:07 dist-test-slave-fl7g krb5kdc[3404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:56:07 dist-test-slave-fl7g krb5kdc[3404](info): set up 2 sockets
Sep 26 20:56:07 dist-test-slave-fl7g krb5kdc[3404](info): commencing operation
krb5kdc: starting...
W20260926 20:56:08.376374 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:56:08 dist-test-slave-fl7g krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:56:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:56:08.508200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37065
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:08.590354 3417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:08.590605 3417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:08.590627 3417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:08.591883 3417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:08.591933 3417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:08.591948 3417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:08.591960 3417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:08.591974 3417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:08.593645 3417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:08.593928 3417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:08.594182 3417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:08.599357 3422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:08.599357 3423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:08.599462 3425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:08.599609 3417 server_base.cc:1034] running on GCE node
I20260926 20:56:08.599838 3417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:08.600137 3417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:08.601300 3417 hybrid_clock.cc:648] HybridClock initialized: now 1790456168601289 us; error 31 us; skew 500 ppm
Sep 26 20:56:08 dist-test-slave-fl7g krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:08.603881 3417 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:56:08.604385 3417 webserver.cc:460] Webserver started at http://127.0.98.62:38891/ using document root <none> and password file <none>
I20260926 20:56:08.604650 3417 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:08.604718 3417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:08.604875 3417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:08.606010 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "a956987a715946fe9a3af169ab98684d"
format_stamp: "Formatted at 2026-09-26 20:56:08 on dist-test-slave-fl7g"
server_key: "170a588f75f7d15518251ee353db313b"
server_key_iv: "48fdeb423668d46990353c510d0dcbf0"
server_key_version: "encryptionkey@0"
I20260926 20:56:08.606395 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "a956987a715946fe9a3af169ab98684d"
format_stamp: "Formatted at 2026-09-26 20:56:08 on dist-test-slave-fl7g"
server_key: "170a588f75f7d15518251ee353db313b"
server_key_iv: "48fdeb423668d46990353c510d0dcbf0"
server_key_version: "encryptionkey@0"
I20260926 20:56:08.607781 3417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:56:08.608620 3432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:08.608798 3417 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:08.608906 3417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "a956987a715946fe9a3af169ab98684d"
format_stamp: "Formatted at 2026-09-26 20:56:08 on dist-test-slave-fl7g"
server_key: "170a588f75f7d15518251ee353db313b"
server_key_iv: "48fdeb423668d46990353c510d0dcbf0"
server_key_version: "encryptionkey@0"
I20260926 20:56:08.608984 3417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:08.626585 3417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:08.628842 3417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:08.629016 3417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:08.633316 3417 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37065
I20260926 20:56:08.633437 3484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37065 every 8 connection(s)
I20260926 20:56:08.633776 3417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:08.634783 3485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:08.637163 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d: Bootstrap starting.
I20260926 20:56:08.637815 3485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:08.638120 3485 log.cc:824] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:08.638887 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d: No bootstrap required, opened a new log
I20260926 20:56:08.640544 3485 raft_consensus.cc:348] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a956987a715946fe9a3af169ab98684d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37065 } }
I20260926 20:56:08.640688 3485 raft_consensus.cc:374] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:08.640710 3485 raft_consensus.cc:729] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a956987a715946fe9a3af169ab98684d, State: Initialized, Role: FOLLOWER
I20260926 20:56:08.640880 3485 consensus_queue.cc:260] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a956987a715946fe9a3af169ab98684d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37065 } }
I20260926 20:56:08.640944 3485 raft_consensus.cc:388] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:08.641002 3485 raft_consensus.cc:482] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:08.641052 3485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:08.641904 3485 raft_consensus.cc:504] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a956987a715946fe9a3af169ab98684d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37065 } }
I20260926 20:56:08.642022 3485 leader_election.cc:302] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a956987a715946fe9a3af169ab98684d; no voters:
I20260926 20:56:08.642210 3485 leader_election.cc:288] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:08.642292 3488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:08.642486 3488 raft_consensus.cc:686] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d [term 1 LEADER]: Becoming Leader. State: Replica: a956987a715946fe9a3af169ab98684d, State: Running, Role: LEADER
I20260926 20:56:08.642505 3485 sys_catalog.cc:564] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:08.642587 3488 consensus_queue.cc:237] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a956987a715946fe9a3af169ab98684d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37065 } }
I20260926 20:56:08.643042 3489 sys_catalog.cc:455] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a956987a715946fe9a3af169ab98684d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a956987a715946fe9a3af169ab98684d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37065 } } }
I20260926 20:56:08.643143 3489 sys_catalog.cc:458] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:08.643337 3490 sys_catalog.cc:455] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a956987a715946fe9a3af169ab98684d. Latest consensus state: current_term: 1 leader_uuid: "a956987a715946fe9a3af169ab98684d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a956987a715946fe9a3af169ab98684d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37065 } } }
I20260926 20:56:08.643399 3490 sys_catalog.cc:458] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:08.643386 3497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:08.643671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 3417
I20260926 20:56:08.643762 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:56:08.643972 392 external_mini_cluster.cc:1428] Setting key 3d2072a55fddfb7f320f34c979f11b11
I20260926 20:56:08.644150 3497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:08.646944 3497 catalog_manager.cc:1347] Generated new cluster ID: 2628aabcae1e4d8e939d81a5f70f7514
I20260926 20:56:08.647001 3497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:56:08 dist-test-slave-fl7g krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:08.655774 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:08.644572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.0.98.62:37065)
0926 20:56:08.645067 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:08.645076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:08.645273 (+ 197us) server_negotiation.cc:407] Connection header received
0926 20:56:08.645404 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:08.645412 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:08.645491 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:08.645656 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:08.646563 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:08.647655 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:08.648546 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:08.649030 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:08.649174 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:08.651880 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:08.651892 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:08.651893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:08.651907 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:08.653875 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:08.654349 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:08.654352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:08.654353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:08.654497 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:08.654743 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:08.654744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:08.654744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:08.655026 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:08.655217 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:08.655304 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 20:56:08.655579 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":64,"threads_started":1}
I20260926 20:56:08.663792 3497 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:08.664373 3497 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:08.672804 3497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a956987a715946fe9a3af169ab98684d: Generated new TSK 0
I20260926 20:56:08.673050 3497 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:56:08.708523 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37065
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:08.793789 3511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:08.794010 3511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:08.794039 3511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:08.795295 3511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:08.795344 3511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:08.795374 3511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:56:08.797036 3511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:08.797255 3511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:08.797452 3511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:56:08.803785 3511 server_base.cc:1034] running on GCE node
W20260926 20:56:08.803715 3517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:08.803715 3519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:08.803794 3516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:08.804210 3511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:08.804559 3511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:08.805727 3511 hybrid_clock.cc:648] HybridClock initialized: now 1790456168805674 us; error 73 us; skew 500 ppm
Sep 26 20:56:08 dist-test-slave-fl7g krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:08.808518 3511 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:56:08.809015 3511 webserver.cc:460] Webserver started at http://127.0.98.1:45457/ using document root <none> and password file <none>
I20260926 20:56:08.809289 3511 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:08.809338 3511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:08.809514 3511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:08.810674 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "f57b0e53f51148ae980ff8475aaecb4f"
format_stamp: "Formatted at 2026-09-26 20:56:08 on dist-test-slave-fl7g"
server_key: "4081de33cec6b82be226347240289228"
server_key_iv: "b697c6d19c975a44fdc25c1e001ad40c"
server_key_version: "encryptionkey@0"
I20260926 20:56:08.811108 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f57b0e53f51148ae980ff8475aaecb4f"
format_stamp: "Formatted at 2026-09-26 20:56:08 on dist-test-slave-fl7g"
server_key: "4081de33cec6b82be226347240289228"
server_key_iv: "b697c6d19c975a44fdc25c1e001ad40c"
server_key_version: "encryptionkey@0"
I20260926 20:56:08.812731 3511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 20:56:08.813701 3526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:08.813892 3511 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:08.813989 3511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "f57b0e53f51148ae980ff8475aaecb4f"
format_stamp: "Formatted at 2026-09-26 20:56:08 on dist-test-slave-fl7g"
server_key: "4081de33cec6b82be226347240289228"
server_key_iv: "b697c6d19c975a44fdc25c1e001ad40c"
server_key_version: "encryptionkey@0"
I20260926 20:56:08.814073 3511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:08.829788 3511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:08.832255 3511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:08.832391 3511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:08.832723 3511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:08.833004 3511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:08.833114 3511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:08.833177 3511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:08.833194 3511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:08.839399 3511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42533
I20260926 20:56:08.839435 3639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42533 every 8 connection(s)
I20260926 20:56:08.839785 3511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:56:08.844005 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 3511
I20260926 20:56:08.844115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:56:08.844309 392 external_mini_cluster.cc:1428] Setting key 6aabf419e4ec9201c80c1e586a02b802
Sep 26 20:56:08 dist-test-slave-fl7g krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:08.851248 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:08.840554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39513 (local address 127.0.98.62:37065)
0926 20:56:08.840600 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:08.840602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:08.840844 (+ 242us) server_negotiation.cc:407] Connection header received
0926 20:56:08.841414 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:08.841416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:08.841429 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:08.841524 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:08.842579 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:08.843105 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:08.844576 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:08.844843 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:08.844936 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:08.847833 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:08.847836 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:08.847836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:08.847847 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:08.849582 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:08.850109 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:08.850111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:08.850111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:08.850184 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:08.850494 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:08.850495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:08.850495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:08.850633 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:08.850722 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:08.851040 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 20:56:08.851144 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:56:08.851472 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:08.840555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37065 (local address 127.0.98.1:39513)
0926 20:56:08.840713 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:08.840752 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:08.841292 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:08.841615 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:08.841619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:08.842238 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:08.842467 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:08.842474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:08.843208 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:08.843209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:08.844480 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:08.844482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:08.844946 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:08.844947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:08.845087 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:08.845685 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:08.845696 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:08.847441 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:08.849698 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:08.849709 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:08.849712 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:08.850000 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:08.850276 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:08.850276 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:08.850277 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:08.850415 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:08.850709 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:08.850717 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:08.850885 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 20:56:08.851016 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 20:56:08.851201 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260926 20:56:08.852094 3640 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37065
I20260926 20:56:08.852233 3640 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:08.852506 3640 heartbeater.cc:511] Master 127.0.98.62:37065 requested a full tablet report, sending...
I20260926 20:56:08.853165 3449 ts_manager.cc:194] Registered new tserver with Master: f57b0e53f51148ae980ff8475aaecb4f (127.0.98.1:42533)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 20:56:08.853933 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:56:08.875290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37065
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:08.963918 3645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:08.964190 3645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:08.964231 3645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:08.965533 3645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:08.965622 3645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:08.965669 3645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:56:08.967249 3645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:08.967607 3645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:08.967868 3645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:08.973099 3650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:08.973089 3651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:08.973089 3653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:08.973367 3645 server_base.cc:1034] running on GCE node
I20260926 20:56:08.973567 3645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:08.973850 3645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:08.975032 3645 hybrid_clock.cc:648] HybridClock initialized: now 1790456168975020 us; error 32 us; skew 500 ppm
Sep 26 20:56:08 dist-test-slave-fl7g krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:08.977754 3645 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:56:08.978247 3645 webserver.cc:460] Webserver started at http://127.0.98.2:34585/ using document root <none> and password file <none>
I20260926 20:56:08.978502 3645 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:08.978575 3645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:08.978734 3645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:08.979841 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "64f5d48b09b944f38a87f3a9e646e948"
format_stamp: "Formatted at 2026-09-26 20:56:08 on dist-test-slave-fl7g"
server_key: "4cc897d40b3b23ca1ae5a824df9c1b38"
server_key_iv: "a8f748b33800a8d3b15d225c2845fafc"
server_key_version: "encryptionkey@0"
I20260926 20:56:08.980219 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "64f5d48b09b944f38a87f3a9e646e948"
format_stamp: "Formatted at 2026-09-26 20:56:08 on dist-test-slave-fl7g"
server_key: "4cc897d40b3b23ca1ae5a824df9c1b38"
server_key_iv: "a8f748b33800a8d3b15d225c2845fafc"
server_key_version: "encryptionkey@0"
I20260926 20:56:08.981762 3645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:08.982533 3660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:08.982733 3645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:08.982826 3645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "64f5d48b09b944f38a87f3a9e646e948"
format_stamp: "Formatted at 2026-09-26 20:56:08 on dist-test-slave-fl7g"
server_key: "4cc897d40b3b23ca1ae5a824df9c1b38"
server_key_iv: "a8f748b33800a8d3b15d225c2845fafc"
server_key_version: "encryptionkey@0"
I20260926 20:56:08.982899 3645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:09.021659 3645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:09.024242 3645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:09.024405 3645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:09.024713 3645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:09.025053 3645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:09.025089 3645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:09.025112 3645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:09.025146 3645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:09.030941 3645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42327
I20260926 20:56:09.031011 3773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42327 every 8 connection(s)
I20260926 20:56:09.031322 3645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:56:09 dist-test-slave-fl7g krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:09.041217 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:09.032066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40689 (local address 127.0.98.62:37065)
0926 20:56:09.032140 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:09.032143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:09.032341 (+ 198us) server_negotiation.cc:407] Connection header received
0926 20:56:09.032931 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:09.032932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:09.032945 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:09.033001 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:09.033996 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:09.034502 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:09.035569 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:09.035765 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:09.035872 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:09.038392 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:09.038394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:09.038394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:09.038402 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:09.039851 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:09.040258 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:09.040259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:09.040259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:09.040332 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:09.040620 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:09.040621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:09.040621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:09.040744 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:09.040807 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:09.041059 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 20:56:09.041137 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:56:09.041266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 3645
I20260926 20:56:09.041368 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:56:09.041553 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:09.032039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37065 (local address 127.0.98.2:40689)
0926 20:56:09.032201 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:09.032250 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:09.032811 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:09.033086 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:09.033090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:09.033672 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:09.033871 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:09.033875 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:09.034609 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:09.034610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:09.035471 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:09.035473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:09.035881 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:09.035882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:09.036024 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:09.036652 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:09.036660 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:09.038269 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:09.039942 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:09.039948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:09.039950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:09.040181 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:09.040396 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:09.040397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:09.040397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:09.040556 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:09.040805 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:09.040809 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:09.040931 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 20:56:09.041012 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 20:56:09.041247 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260926 20:56:09.041577 392 external_mini_cluster.cc:1428] Setting key 66e2bdfe211109e030cf820ef5b63112
I20260926 20:56:09.042106 3774 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37065
I20260926 20:56:09.042235 3774 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:09.042446 3774 heartbeater.cc:511] Master 127.0.98.62:37065 requested a full tablet report, sending...
I20260926 20:56:09.042948 3448 ts_manager.cc:194] Registered new tserver with Master: 64f5d48b09b944f38a87f3a9e646e948 (127.0.98.2:42327)
I20260926 20:56:09.043380 3448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40689
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:56:09.071811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37065
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:09.155776 3779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:09.156061 3779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:09.156107 3779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:09.157434 3779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:09.157528 3779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:09.157578 3779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:56:09.159153 3779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:09.159459 3779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:09.159722 3779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:09.165036 3784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:09.165221 3779 server_base.cc:1034] running on GCE node
W20260926 20:56:09.165040 3785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:09.165040 3787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:09.165577 3779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:09.165882 3779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:09.167042 3779 hybrid_clock.cc:648] HybridClock initialized: now 1790456169167023 us; error 39 us; skew 500 ppm
Sep 26 20:56:09 dist-test-slave-fl7g krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:09.170312 3779 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:56:09.170866 3779 webserver.cc:460] Webserver started at http://127.0.98.3:46191/ using document root <none> and password file <none>
I20260926 20:56:09.171101 3779 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:09.171149 3779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:09.171303 3779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:09.172544 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "36905216d1e6419db407e1b8876dee2a"
format_stamp: "Formatted at 2026-09-26 20:56:09 on dist-test-slave-fl7g"
server_key: "da6ef4ee4e419c5a1b38cdbbd2d0b715"
server_key_iv: "9685d16c90e4669ff496c9b616327efe"
server_key_version: "encryptionkey@0"
I20260926 20:56:09.172968 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "36905216d1e6419db407e1b8876dee2a"
format_stamp: "Formatted at 2026-09-26 20:56:09 on dist-test-slave-fl7g"
server_key: "da6ef4ee4e419c5a1b38cdbbd2d0b715"
server_key_iv: "9685d16c90e4669ff496c9b616327efe"
server_key_version: "encryptionkey@0"
I20260926 20:56:09.174566 3779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:56:09.175426 3794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:09.175771 3779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:09.175900 3779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "36905216d1e6419db407e1b8876dee2a"
format_stamp: "Formatted at 2026-09-26 20:56:09 on dist-test-slave-fl7g"
server_key: "da6ef4ee4e419c5a1b38cdbbd2d0b715"
server_key_iv: "9685d16c90e4669ff496c9b616327efe"
server_key_version: "encryptionkey@0"
I20260926 20:56:09.175993 3779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:09.204511 3779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:09.207020 3779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:09.207168 3779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:09.207448 3779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:09.207783 3779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:09.207840 3779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:09.207883 3779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:09.207911 3779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:09.214044 3779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42301
I20260926 20:56:09.214089 3907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42301 every 8 connection(s)
I20260926 20:56:09.214434 3779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:56:09 dist-test-slave-fl7g krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:09.222636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 3779
I20260926 20:56:09.222747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:56:09.222971 392 external_mini_cluster.cc:1428] Setting key f044dec4646bb6703112e791f8fa9d3f
I20260926 20:56:09.226349 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:09.215274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46877 (local address 127.0.98.62:37065)
0926 20:56:09.215347 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:09.215349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:09.215653 (+ 304us) server_negotiation.cc:407] Connection header received
0926 20:56:09.216530 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:09.216532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:09.216545 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:09.216613 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:09.217949 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:09.218452 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:09.220079 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:09.220282 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:09.220348 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:09.223148 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:09.223151 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:09.223151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:09.223159 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:09.224892 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:09.225331 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:09.225333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:09.225333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:09.225402 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:09.225672 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:09.225673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:09.225673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:09.225803 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:09.225879 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:09.226171 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 20:56:09.226252 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:56:09.226596 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:09.215266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37065 (local address 127.0.98.3:46877)
0926 20:56:09.215476 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:09.215542 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:09.216325 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:09.216696 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:09.216701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:09.217534 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:09.217840 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:09.217847 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:09.218557 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:09.218558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:09.219961 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:09.219964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:09.220373 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:09.220375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:09.220564 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:09.221114 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:09.221125 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:09.222993 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:09.224996 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:09.225005 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:09.225007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:09.225250 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:09.225470 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:09.225471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:09.225472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:09.225600 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:09.225870 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:09.225876 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:09.226029 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 20:56:09.226140 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 20:56:09.226315 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":86,"threads_started":1}
I20260926 20:56:09.227049 3908 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37065
I20260926 20:56:09.227174 3908 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:09.227394 3908 heartbeater.cc:511] Master 127.0.98.62:37065 requested a full tablet report, sending...
I20260926 20:56:09.227831 3448 ts_manager.cc:194] Registered new tserver with Master: 36905216d1e6419db407e1b8876dee2a (127.0.98.3:42301)
I20260926 20:56:09.228399 3448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46877
I20260926 20:56:09.234951 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:56:09 dist-test-slave-fl7g krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456169, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 20:56:09 dist-test-slave-fl7g krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456169, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:09.265978 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:09.256482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.0.98.62:37065)
0926 20:56:09.256535 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:09.256537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:09.256548 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:56:09.256632 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:09.256634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:09.256648 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:09.256719 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:09.257524 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:09.258355 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:09.259488 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:09.259837 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:09.259919 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:09.262815 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:09.262817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:09.262818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:09.262827 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:09.264553 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:09.264939 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:09.264940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:09.264940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:09.265005 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:09.265233 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:09.265234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:09.265235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:09.265447 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:09.265532 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:09.265774 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 20:56:09.265866 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 20:56:09.266215 3448 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42984
I20260926 20:56:09.266659 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 3511
I20260926 20:56:09.273054 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 3645
I20260926 20:56:09.278658 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 3779
I20260926 20:56:09.283479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 3417
2026-09-26T20:56:09Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1952 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:56:09 dist-test-slave-fl7g krb5kdc[3920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:56:09 dist-test-slave-fl7g krb5kdc[3920](info): set up 2 sockets
Sep 26 20:56:09 dist-test-slave-fl7g krb5kdc[3920](info): commencing operation
krb5kdc: starting...
W20260926 20:56:10.324532 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:56:10 dist-test-slave-fl7g krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:56:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:56:10.453084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37389
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:10.536696 3933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:10.536944 3933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:10.536970 3933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:10.538185 3933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:10.538224 3933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:10.538239 3933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:10.538252 3933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:10.538264 3933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:10.539731 3933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:10.539944 3933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:10.540127 3933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:10.545356 3938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:10.545354 3941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:10.545370 3939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:10.545629 3933 server_base.cc:1034] running on GCE node
I20260926 20:56:10.545874 3933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:10.546205 3933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:10.547389 3933 hybrid_clock.cc:648] HybridClock initialized: now 1790456170547379 us; error 36 us; skew 500 ppm
Sep 26 20:56:10 dist-test-slave-fl7g krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:10.553403 3933 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:56:10.554009 3933 webserver.cc:460] Webserver started at http://127.0.98.62:43401/ using document root <none> and password file <none>
I20260926 20:56:10.554293 3933 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:10.554346 3933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:10.554534 3933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:10.555702 3933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "1124594e4ed34af0a58edbca324d675d"
format_stamp: "Formatted at 2026-09-26 20:56:10 on dist-test-slave-fl7g"
server_key: "bcbfd7a1fdd7a13ed8e9c41c04f2aa18"
server_key_iv: "62d20c6bfed6619f796f9f56a51b2e6e"
server_key_version: "encryptionkey@0"
I20260926 20:56:10.556102 3933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "1124594e4ed34af0a58edbca324d675d"
format_stamp: "Formatted at 2026-09-26 20:56:10 on dist-test-slave-fl7g"
server_key: "bcbfd7a1fdd7a13ed8e9c41c04f2aa18"
server_key_iv: "62d20c6bfed6619f796f9f56a51b2e6e"
server_key_version: "encryptionkey@0"
I20260926 20:56:10.557739 3933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:56:10.558657 3948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:10.558876 3933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:10.558957 3933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "1124594e4ed34af0a58edbca324d675d"
format_stamp: "Formatted at 2026-09-26 20:56:10 on dist-test-slave-fl7g"
server_key: "bcbfd7a1fdd7a13ed8e9c41c04f2aa18"
server_key_iv: "62d20c6bfed6619f796f9f56a51b2e6e"
server_key_version: "encryptionkey@0"
I20260926 20:56:10.559072 3933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:10.580281 3933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:10.582662 3933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:10.582849 3933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:10.587206 3933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37389
I20260926 20:56:10.587240 4000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37389 every 8 connection(s)
I20260926 20:56:10.587622 3933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:10.588611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 3933
I20260926 20:56:10.588701 4001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:10.588733 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:56:10.588930 392 external_mini_cluster.cc:1428] Setting key 9695fd8bd7fd8b14f2c3ee362ed88032
I20260926 20:56:10.592149 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d: Bootstrap starting.
I20260926 20:56:10.592855 4001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:10.593160 4001 log.cc:824] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:10.593878 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d: No bootstrap required, opened a new log
Sep 26 20:56:10 dist-test-slave-fl7g krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:10.596205 4001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1124594e4ed34af0a58edbca324d675d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37389 } }
I20260926 20:56:10.596361 4001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:10.596383 4001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1124594e4ed34af0a58edbca324d675d, State: Initialized, Role: FOLLOWER
I20260926 20:56:10.596637 4001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1124594e4ed34af0a58edbca324d675d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37389 } }
I20260926 20:56:10.596710 4001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:10.596735 4001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:10.596791 4001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:10.597399 4001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1124594e4ed34af0a58edbca324d675d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37389 } }
I20260926 20:56:10.597524 4001 leader_election.cc:302] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1124594e4ed34af0a58edbca324d675d; no voters:
I20260926 20:56:10.597735 4001 leader_election.cc:288] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:10.597823 4006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:10.598028 4001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:10.598955 4006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d [term 1 LEADER]: Becoming Leader. State: Replica: 1124594e4ed34af0a58edbca324d675d, State: Running, Role: LEADER
I20260926 20:56:10.599068 4006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1124594e4ed34af0a58edbca324d675d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37389 } }
I20260926 20:56:10.599507 4006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1124594e4ed34af0a58edbca324d675d. Latest consensus state: current_term: 1 leader_uuid: "1124594e4ed34af0a58edbca324d675d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1124594e4ed34af0a58edbca324d675d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37389 } } }
I20260926 20:56:10.599851 4007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1124594e4ed34af0a58edbca324d675d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1124594e4ed34af0a58edbca324d675d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37389 } } }
I20260926 20:56:10.599962 4006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:10.600042 4007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d [sys.catalog]: This master's current role is: LEADER
W20260926 20:56:10.600962 4021 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:56:10.601089 4021 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:56:10.601106 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:10.589593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.0.98.62:37389)
0926 20:56:10.589780 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:10.589788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:10.589815 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:56:10.590280 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:10.590287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:10.590339 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:10.590463 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:10.591301 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:10.592194 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:10.593119 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:10.593607 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:10.593751 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:10.596622 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:10.596630 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:10.596630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:10.596648 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:10.598685 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:10.599147 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:10.599151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:10.599151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:10.599275 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:10.599555 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:10.599556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:10.599557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:10.599924 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:10.600120 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:10.600652 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 20:56:10.600843 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260926 20:56:10.601151 4019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:10.601310 4019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:10.602952 4019 catalog_manager.cc:1347] Generated new cluster ID: 430da121c8584f14a496db4ff62c38ec
I20260926 20:56:10.603014 4019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:56:10.619592 4019 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:10.620182 4019 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:10.627082 4019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1124594e4ed34af0a58edbca324d675d: Generated new TSK 0
I20260926 20:56:10.627341 4019 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:56:10.659837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37389
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:10.745146 4027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:10.745412 4027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:10.745458 4027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:10.746703 4027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:10.746783 4027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:10.746832 4027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:56:10.748401 4027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:10.748719 4027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:10.748972 4027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:10.754168 4032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:10.754197 4033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:10.754160 4035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:10.754567 4027 server_base.cc:1034] running on GCE node
I20260926 20:56:10.754799 4027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:10.755066 4027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:10.756253 4027 hybrid_clock.cc:648] HybridClock initialized: now 1790456170756242 us; error 43 us; skew 500 ppm
Sep 26 20:56:10 dist-test-slave-fl7g krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:10.759117 4027 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:56:10.759744 4027 webserver.cc:460] Webserver started at http://127.0.98.1:37083/ using document root <none> and password file <none>
I20260926 20:56:10.759994 4027 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:10.760066 4027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:10.760215 4027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:10.761425 4027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "690a94241948490fb05eef7095ca64ea"
format_stamp: "Formatted at 2026-09-26 20:56:10 on dist-test-slave-fl7g"
server_key: "fcc56e691e506d07b4aecfcf17e8822d"
server_key_iv: "7547ff576f917211f2e8d83982366533"
server_key_version: "encryptionkey@0"
I20260926 20:56:10.761818 4027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "690a94241948490fb05eef7095ca64ea"
format_stamp: "Formatted at 2026-09-26 20:56:10 on dist-test-slave-fl7g"
server_key: "fcc56e691e506d07b4aecfcf17e8822d"
server_key_iv: "7547ff576f917211f2e8d83982366533"
server_key_version: "encryptionkey@0"
I20260926 20:56:10.763214 4027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:56:10.764019 4042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:10.764207 4027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:10.764303 4027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "690a94241948490fb05eef7095ca64ea"
format_stamp: "Formatted at 2026-09-26 20:56:10 on dist-test-slave-fl7g"
server_key: "fcc56e691e506d07b4aecfcf17e8822d"
server_key_iv: "7547ff576f917211f2e8d83982366533"
server_key_version: "encryptionkey@0"
I20260926 20:56:10.764385 4027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:10.780332 4027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:10.782886 4027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:10.783080 4027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:10.783340 4027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:10.783691 4027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:10.783753 4027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:10.783795 4027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:10.783839 4027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:10.790105 4027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35481
I20260926 20:56:10.790160 4155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35481 every 8 connection(s)
I20260926 20:56:10.790529 4027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:56:10.796069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 4027
I20260926 20:56:10.796238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:56:10.796550 392 external_mini_cluster.cc:1428] Setting key d6ef4443347a472d9e84e5e53dc2a807
Sep 26 20:56:10 dist-test-slave-fl7g krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:10.801756 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:10.791369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52237 (local address 127.0.98.62:37389)
0926 20:56:10.791469 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:10.791472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:10.791707 (+ 235us) server_negotiation.cc:407] Connection header received
0926 20:56:10.792785 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:10.792787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:10.792800 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:10.792854 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:10.793533 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:10.794054 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:10.795112 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:10.795335 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:10.795415 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:10.798105 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:10.798107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:10.798107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:10.798114 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:10.800057 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:10.800513 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:10.800515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:10.800515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:10.800591 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:10.800886 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:10.800887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:10.800887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:10.801070 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:10.801160 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:10.801532 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 20:56:10.801640 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:56:10.801782 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:10.791272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37389 (local address 127.0.98.1:52237)
0926 20:56:10.791566 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:10.791606 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:10.792660 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:10.792934 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:10.792941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:10.793225 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:10.793425 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:10.793430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:10.794155 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:10.794156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:10.795004 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:10.795007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:10.795435 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:10.795436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:10.795603 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:10.796380 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:10.796389 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:10.797994 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:10.800157 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:10.800165 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:10.800168 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:10.800448 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:10.800675 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:10.800676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:10.800677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:10.800798 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:10.801187 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:10.801197 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:10.801365 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 20:56:10.801459 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:56:10.801609 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
I20260926 20:56:10.802781 4156 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37389
I20260926 20:56:10.802901 4156 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:10.803126 4156 heartbeater.cc:511] Master 127.0.98.62:37389 requested a full tablet report, sending...
I20260926 20:56:10.803691 3965 ts_manager.cc:194] Registered new tserver with Master: 690a94241948490fb05eef7095ca64ea (127.0.98.1:35481)
I20260926 20:56:10.804325 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52237
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:56:10.827409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37389
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:10.913163 4161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:10.913383 4161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:10.913407 4161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:10.914559 4161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:10.914605 4161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:10.914633 4161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:56:10.916114 4161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:10.916347 4161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:10.916627 4161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:10.922125 4166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:10.922129 4167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:10.922125 4169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:10.922286 4161 server_base.cc:1034] running on GCE node
I20260926 20:56:10.922572 4161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:10.922897 4161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:10.924103 4161 hybrid_clock.cc:648] HybridClock initialized: now 1790456170924100 us; error 43 us; skew 500 ppm
Sep 26 20:56:10 dist-test-slave-fl7g krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:10.927047 4161 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:56:10.927667 4161 webserver.cc:460] Webserver started at http://127.0.98.2:35579/ using document root <none> and password file <none>
I20260926 20:56:10.927918 4161 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:10.928002 4161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:10.928167 4161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:10.929378 4161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "664e6aa86a934dfd9ad059cab0aae453"
format_stamp: "Formatted at 2026-09-26 20:56:10 on dist-test-slave-fl7g"
server_key: "7a666fa7d1aeacfd10d053ee23a8df78"
server_key_iv: "81b7d07aeedf51780e3ecc3a4e88c455"
server_key_version: "encryptionkey@0"
I20260926 20:56:10.929785 4161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "664e6aa86a934dfd9ad059cab0aae453"
format_stamp: "Formatted at 2026-09-26 20:56:10 on dist-test-slave-fl7g"
server_key: "7a666fa7d1aeacfd10d053ee23a8df78"
server_key_iv: "81b7d07aeedf51780e3ecc3a4e88c455"
server_key_version: "encryptionkey@0"
I20260926 20:56:10.931219 4161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:56:10.932124 4176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:10.932355 4161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:10.932533 4161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "664e6aa86a934dfd9ad059cab0aae453"
format_stamp: "Formatted at 2026-09-26 20:56:10 on dist-test-slave-fl7g"
server_key: "7a666fa7d1aeacfd10d053ee23a8df78"
server_key_iv: "81b7d07aeedf51780e3ecc3a4e88c455"
server_key_version: "encryptionkey@0"
I20260926 20:56:10.932610 4161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:10.961423 4161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:10.964222 4161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:10.964362 4161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:10.964692 4161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:10.965070 4161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:10.965108 4161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:10.965160 4161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:10.965190 4161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:10.971884 4161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39773
I20260926 20:56:10.971931 4289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39773 every 8 connection(s)
I20260926 20:56:10.972302 4161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:56:10.972968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 4161
I20260926 20:56:10.973099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:56:10.973340 392 external_mini_cluster.cc:1428] Setting key 504c458dfb8486d73afa79c40982f552
Sep 26 20:56:10 dist-test-slave-fl7g krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 20:56:10.985270 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:10.973300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52917 (local address 127.0.98.62:37389)
0926 20:56:10.973403 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:10.973406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:10.973645 (+ 239us) server_negotiation.cc:407] Connection header received
0926 20:56:10.974986 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:10.974987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:10.974998 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:10.975057 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:10.976022 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:10.976858 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:10.978592 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:10.978790 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:10.978858 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:10.981439 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:10.981441 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:10.981441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:10.981450 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:10.983091 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:10.983635 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:10.983637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:10.983637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:10.983709 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:10.984317 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:10.984318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:10.984318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:10.984562 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:10.984633 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:10.985081 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 20:56:10.985193 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:56:10.985417 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:10.973288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37389 (local address 127.0.98.2:52917)
0926 20:56:10.973504 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:10.973565 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:10.974862 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:10.975138 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:10.975142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:10.975591 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:10.975899 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:10.975905 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:10.976980 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:10.976981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:10.978466 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:10.978470 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:10.978891 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:10.978892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:10.979067 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:10.979786 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:10.979795 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:10.981325 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:10.983210 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:10.983218 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:10.983221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:10.983541 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:10.983879 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:10.983880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:10.983914 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:10.984207 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:10.984637 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:10.984644 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:10.984807 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 20:56:10.985037 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 20:56:10.985202 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 20:56:10.986014 4290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37389
I20260926 20:56:10.986166 4290 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:10.986392 4290 heartbeater.cc:511] Master 127.0.98.62:37389 requested a full tablet report, sending...
I20260926 20:56:10.986912 3965 ts_manager.cc:194] Registered new tserver with Master: 664e6aa86a934dfd9ad059cab0aae453 (127.0.98.2:39773)
I20260926 20:56:10.987356 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:56:11.004977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37389
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:11.088663 4295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:11.088900 4295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:11.088927 4295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:11.090207 4295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:11.090260 4295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:11.090289 4295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:56:11.091786 4295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:11.092015 4295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:11.092200 4295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:11.097358 4301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:11.097361 4300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:11.097378 4303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:11.097883 4295 server_base.cc:1034] running on GCE node
I20260926 20:56:11.098199 4295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:11.098522 4295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:11.099684 4295 hybrid_clock.cc:648] HybridClock initialized: now 1790456171099657 us; error 49 us; skew 500 ppm
Sep 26 20:56:11 dist-test-slave-fl7g krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:11.102593 4295 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:56:11.103060 4295 webserver.cc:460] Webserver started at http://127.0.98.3:35683/ using document root <none> and password file <none>
I20260926 20:56:11.103260 4295 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:11.103302 4295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:11.103389 4295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:11.104480 4295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "6ea0baac75ea4a7bbd22d472ce83a159"
format_stamp: "Formatted at 2026-09-26 20:56:11 on dist-test-slave-fl7g"
server_key: "b8c1e2c1cd66650b8c0516b6da588493"
server_key_iv: "b3bdedf65a93180786a18491a92db554"
server_key_version: "encryptionkey@0"
I20260926 20:56:11.104831 4295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6ea0baac75ea4a7bbd22d472ce83a159"
format_stamp: "Formatted at 2026-09-26 20:56:11 on dist-test-slave-fl7g"
server_key: "b8c1e2c1cd66650b8c0516b6da588493"
server_key_iv: "b3bdedf65a93180786a18491a92db554"
server_key_version: "encryptionkey@0"
I20260926 20:56:11.106151 4295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:11.106933 4310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:11.107195 4295 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:11.107286 4295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "6ea0baac75ea4a7bbd22d472ce83a159"
format_stamp: "Formatted at 2026-09-26 20:56:11 on dist-test-slave-fl7g"
server_key: "b8c1e2c1cd66650b8c0516b6da588493"
server_key_iv: "b3bdedf65a93180786a18491a92db554"
server_key_version: "encryptionkey@0"
I20260926 20:56:11.107349 4295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:11.119594 4295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:11.122217 4295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:11.122344 4295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:11.122563 4295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:11.122891 4295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:11.122925 4295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:11.122951 4295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:11.122996 4295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:11.129177 4295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38701
I20260926 20:56:11.129287 4423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38701 every 8 connection(s)
I20260926 20:56:11.129607 4295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:56:11.130155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 4295
I20260926 20:56:11.130235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:56:11.130451 392 external_mini_cluster.cc:1428] Setting key 92ebc8ebe74c4f21a62f3c9cf072aeb9
Sep 26 20:56:11 dist-test-slave-fl7g krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:11.139894 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:11.130461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33181 (local address 127.0.98.62:37389)
0926 20:56:11.130551 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:11.130554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:11.130791 (+ 237us) server_negotiation.cc:407] Connection header received
0926 20:56:11.131971 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:11.131972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:11.131982 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:11.132033 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:11.132754 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:11.133249 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:11.134257 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:11.134454 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:11.134510 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:11.137036 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:11.137037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:11.137038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:11.137046 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:11.138591 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:11.139032 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:11.139033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:11.139033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:11.139086 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:11.139316 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:11.139317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:11.139317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:11.139423 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:11.139475 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:11.139751 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 20:56:11.139825 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:56:11.140139 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:11.130447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37389 (local address 127.0.98.3:33181)
0926 20:56:11.130658 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:11.130710 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:11.131857 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:11.132094 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:11.132098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:11.132363 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:11.132661 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:11.132666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:11.133333 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:11.133334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:11.134174 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:11.134176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:11.134598 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:11.134599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:11.134738 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:11.135302 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:11.135311 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:11.136933 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:11.138692 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:11.138699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:11.138701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:11.138953 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:11.139142 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:11.139143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:11.139143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:11.139261 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:11.139497 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:11.139502 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:11.139634 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 20:56:11.139723 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:56:11.139852 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":106,"threads_started":1}
I20260926 20:56:11.140811 4424 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37389
I20260926 20:56:11.140919 4424 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:11.141134 4424 heartbeater.cc:511] Master 127.0.98.62:37389 requested a full tablet report, sending...
I20260926 20:56:11.141583 3965 ts_manager.cc:194] Registered new tserver with Master: 6ea0baac75ea4a7bbd22d472ce83a159 (127.0.98.3:38701)
I20260926 20:56:11.142158 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33181
I20260926 20:56:11.142537 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:56:11 dist-test-slave-fl7g krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:56:11 dist-test-slave-fl7g krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:56:11.167390 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:11.158580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58712 (local address 127.0.98.1:35481)
0926 20:56:11.158770 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:11.158778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:11.158797 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:56:11.158824 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:11.158827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:11.158841 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:11.158922 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:11.159610 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:11.160346 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:11.161213 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:11.161647 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:11.161715 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:11.164176 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:11.164187 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:11.164187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:11.164202 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:11.165865 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:11.166362 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:11.166367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:11.166367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:11.166498 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:11.166732 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:11.166733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:11.166733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:11.166901 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:11.167077 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:11.167152 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 20:56:11.167248 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":115,"threads_started":1}
Sep 26 20:56:11 dist-test-slave-fl7g krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:11.176296 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:11.168904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55062 (local address 127.0.98.62:37389)
0926 20:56:11.168983 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:11.168985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:11.169003 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:56:11.169047 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:11.169047 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:11.169056 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:11.169109 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:11.169601 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:11.170158 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:11.170920 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:11.171138 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:11.171201 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:11.173660 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:11.173662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:11.173662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:11.173669 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:11.175113 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:11.175466 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:11.175467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:11.175468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:11.175543 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:11.175769 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:11.175770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:11.175770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:11.175913 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:11.175985 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:11.176136 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 20:56:11.176207 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 20:56:11.176648 3964 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55062
I20260926 20:56:11.177053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 4027
I20260926 20:56:11.181497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 4161
I20260926 20:56:11.185945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 4295
I20260926 20:56:11.190356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 3933
2026-09-26T20:56:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1905 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:56:11 dist-test-slave-fl7g krb5kdc[4436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:56:11 dist-test-slave-fl7g krb5kdc[4436](info): set up 2 sockets
Sep 26 20:56:11 dist-test-slave-fl7g krb5kdc[4436](info): commencing operation
krb5kdc: starting...
W20260926 20:56:12.228338 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:56:12 dist-test-slave-fl7g krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:56:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:56:12.356544 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41029
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:12.441442 4449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:12.441679 4449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:12.441704 4449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:12.443017 4449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:12.443060 4449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:12.443075 4449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:12.443087 4449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:12.443100 4449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:12.444711 4449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:12.444985 4449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:12.445240 4449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:56:12.450623 4449 server_base.cc:1034] running on GCE node
W20260926 20:56:12.451273 4454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:12.451248 4455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:12.451306 4457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:12.451781 4449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:12.452126 4449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:12.453281 4449 hybrid_clock.cc:648] HybridClock initialized: now 1790456172453268 us; error 31 us; skew 500 ppm
Sep 26 20:56:12 dist-test-slave-fl7g krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:12.455986 4449 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:56:12.456331 4449 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:12.456378 4449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:12.456576 4449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:12.457690 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "3da1a0d7e7b541ac909a783a7ad5181f"
format_stamp: "Formatted at 2026-09-26 20:56:12 on dist-test-slave-fl7g"
server_key: "d46b3a257b1e6cefa4d77da09a912de8"
server_key_iv: "3d2fc105db8cb5860d17d07f4bb57908"
server_key_version: "encryptionkey@0"
I20260926 20:56:12.458017 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "3da1a0d7e7b541ac909a783a7ad5181f"
format_stamp: "Formatted at 2026-09-26 20:56:12 on dist-test-slave-fl7g"
server_key: "d46b3a257b1e6cefa4d77da09a912de8"
server_key_iv: "3d2fc105db8cb5860d17d07f4bb57908"
server_key_version: "encryptionkey@0"
I20260926 20:56:12.459491 4449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:56:12.460321 4463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:12.460625 4449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:12.460728 4449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "3da1a0d7e7b541ac909a783a7ad5181f"
format_stamp: "Formatted at 2026-09-26 20:56:12 on dist-test-slave-fl7g"
server_key: "d46b3a257b1e6cefa4d77da09a912de8"
server_key_iv: "3d2fc105db8cb5860d17d07f4bb57908"
server_key_version: "encryptionkey@0"
I20260926 20:56:12.460800 4449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:12.472157 4449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:12.474390 4449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:12.474525 4449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:12.478699 4449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41029
I20260926 20:56:12.478757 4515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41029 every 8 connection(s)
I20260926 20:56:12.479065 4449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:12.480059 4516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:12.481860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 4449
I20260926 20:56:12.481971 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:56:12.482167 392 external_mini_cluster.cc:1428] Setting key fe41100f513446c58efd578ab0bb07c2
I20260926 20:56:12.482959 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f: Bootstrap starting.
I20260926 20:56:12.483593 4516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:12.483958 4516 log.cc:824] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:12.484778 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f: No bootstrap required, opened a new log
I20260926 20:56:12.487126 4516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da1a0d7e7b541ac909a783a7ad5181f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } }
I20260926 20:56:12.487349 4516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:12.487390 4516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da1a0d7e7b541ac909a783a7ad5181f, State: Initialized, Role: FOLLOWER
I20260926 20:56:12.487533 4516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da1a0d7e7b541ac909a783a7ad5181f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } }
I20260926 20:56:12.487617 4516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:12.487653 4516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:12.487715 4516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:12.488596 4516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da1a0d7e7b541ac909a783a7ad5181f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } }
I20260926 20:56:12.488722 4516 leader_election.cc:302] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3da1a0d7e7b541ac909a783a7ad5181f; no voters:
I20260926 20:56:12.488907 4516 leader_election.cc:288] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:12.488974 4521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:12.489180 4521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f [term 1 LEADER]: Becoming Leader. State: Replica: 3da1a0d7e7b541ac909a783a7ad5181f, State: Running, Role: LEADER
I20260926 20:56:12.489217 4516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 20:56:12 dist-test-slave-fl7g krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:12.489948 4521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da1a0d7e7b541ac909a783a7ad5181f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } }
I20260926 20:56:12.490548 4522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3da1a0d7e7b541ac909a783a7ad5181f. Latest consensus state: current_term: 1 leader_uuid: "3da1a0d7e7b541ac909a783a7ad5181f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da1a0d7e7b541ac909a783a7ad5181f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } }
I20260926 20:56:12.490713 4522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:12.491379 4523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3da1a0d7e7b541ac909a783a7ad5181f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da1a0d7e7b541ac909a783a7ad5181f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } }
I20260926 20:56:12.491452 4523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f [sys.catalog]: This master's current role is: LEADER
W20260926 20:56:12.491891 4535 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:56:12.491954 4535 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:56:12.492551 4537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:12.492874 4537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:12.494555 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:12.482742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42112 (local address 127.0.98.62:41029)
0926 20:56:12.482917 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:12.482928 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:12.482962 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:56:12.483475 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:12.483484 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:12.483537 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:12.483713 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:12.484488 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:12.485546 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:12.486689 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:12.487111 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:12.487204 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:12.490529 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:12.490534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:12.490535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:12.490549 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:12.492858 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:12.493325 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:12.493329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:12.493329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:12.493429 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:12.493653 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:12.493654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:12.493654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:12.493907 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:12.494072 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:12.494190 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 20:56:12.494333 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260926 20:56:12.494589 4537 catalog_manager.cc:1347] Generated new cluster ID: 85ce5602705745489557c693c1abf192
I20260926 20:56:12.494750 4537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:56:12.500946 4537 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:12.501436 4537 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:12.511162 4537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f: Generated new TSK 0
I20260926 20:56:12.511363 4537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:56:12.544205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41029
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:12.629739 4542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:12.629973 4542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:12.630000 4542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:12.631278 4542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:12.631340 4542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:12.631371 4542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:56:12.633085 4542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:12.633358 4542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:12.633631 4542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:12.638900 4548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:12.638916 4547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:12.639048 4550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:12.639289 4542 server_base.cc:1034] running on GCE node
I20260926 20:56:12.639537 4542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:12.639789 4542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:12.641106 4542 hybrid_clock.cc:648] HybridClock initialized: now 1790456172641095 us; error 42 us; skew 500 ppm
Sep 26 20:56:12 dist-test-slave-fl7g krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:12.643968 4542 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:56:12.644345 4542 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:12.644482 4542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:12.644637 4542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:12.645735 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "2eca83aa4c704116af6e0db1d2d8bf59"
format_stamp: "Formatted at 2026-09-26 20:56:12 on dist-test-slave-fl7g"
server_key: "ee23cf3bde150addf30d5738585e7789"
server_key_iv: "3941a2a995ea5e6404d543acfdaca16a"
server_key_version: "encryptionkey@0"
I20260926 20:56:12.646108 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2eca83aa4c704116af6e0db1d2d8bf59"
format_stamp: "Formatted at 2026-09-26 20:56:12 on dist-test-slave-fl7g"
server_key: "ee23cf3bde150addf30d5738585e7789"
server_key_iv: "3941a2a995ea5e6404d543acfdaca16a"
server_key_version: "encryptionkey@0"
I20260926 20:56:12.647517 4542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:56:12.648305 4556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:12.648514 4542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:12.648622 4542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "2eca83aa4c704116af6e0db1d2d8bf59"
format_stamp: "Formatted at 2026-09-26 20:56:12 on dist-test-slave-fl7g"
server_key: "ee23cf3bde150addf30d5738585e7789"
server_key_iv: "3941a2a995ea5e6404d543acfdaca16a"
server_key_version: "encryptionkey@0"
I20260926 20:56:12.648698 4542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:12.666841 4542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:12.669152 4542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:12.669330 4542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:12.669591 4542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:12.669955 4542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:12.670012 4542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:12.670055 4542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:12.670084 4542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:12.676043 4542 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39607
I20260926 20:56:12.676083 4669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39607 every 8 connection(s)
I20260926 20:56:12.676381 4542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:56:12.679971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 4542
I20260926 20:56:12.680063 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:56:12.680272 392 external_mini_cluster.cc:1428] Setting key c409e511f43f20f7d9277d1272745da3
Sep 26 20:56:12 dist-test-slave-fl7g krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:12.687270 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:12.677122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41167 (local address 127.0.98.62:41029)
0926 20:56:12.677185 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:12.677187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:12.677417 (+ 230us) server_negotiation.cc:407] Connection header received
0926 20:56:12.677982 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:12.677984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:12.677997 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:12.678045 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:12.679072 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:12.679590 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:12.680915 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:12.681188 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:12.681264 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:12.683813 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:12.683815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:12.683815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:12.683824 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:12.685613 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:12.686098 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:12.686100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:12.686101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:12.686169 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:12.686484 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:12.686485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:12.686486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:12.686654 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:12.686742 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:12.687057 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 20:56:12.687163 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:56:12.687372 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:12.677095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41029 (local address 127.0.98.1:41167)
0926 20:56:12.677285 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:12.677322 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:12.677858 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:12.678130 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:12.678134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:12.678424 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:12.678941 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:12.678947 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:12.679693 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:12.679694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:12.680847 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:12.680849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:12.681278 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:12.681278 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:12.681395 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:12.681995 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:12.682004 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:12.683702 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:12.685730 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:12.685737 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:12.685739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:12.686011 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:12.686281 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:12.686282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:12.686283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:12.686405 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:12.686752 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:12.686759 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:12.686901 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 20:56:12.687007 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 20:56:12.687146 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260926 20:56:12.688016 4670 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41029
I20260926 20:56:12.688158 4670 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:12.688347 4670 heartbeater.cc:511] Master 127.0.98.62:41029 requested a full tablet report, sending...
I20260926 20:56:12.689057 4480 ts_manager.cc:194] Registered new tserver with Master: 2eca83aa4c704116af6e0db1d2d8bf59 (127.0.98.1:39607)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 20:56:12.689989 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:56:12.708300 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41029
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:12.791414 4675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:12.791640 4675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:12.791664 4675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:12.792902 4675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:12.792949 4675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:12.792977 4675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:56:12.794518 4675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:12.794765 4675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:12.795019 4675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:12.800557 4681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:12.800562 4683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:12.800630 4680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:12.800736 4675 server_base.cc:1034] running on GCE node
I20260926 20:56:12.801103 4675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:12.801328 4675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:12.802501 4675 hybrid_clock.cc:648] HybridClock initialized: now 1790456172802492 us; error 32 us; skew 500 ppm
Sep 26 20:56:12 dist-test-slave-fl7g krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:12.805382 4675 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:56:12.805806 4675 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:12.805861 4675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:12.805968 4675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:12.807092 4675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "20e2e28b57024250838a2987a79857cc"
format_stamp: "Formatted at 2026-09-26 20:56:12 on dist-test-slave-fl7g"
server_key: "20c38bbd5962f42e5b773bd2a28f5868"
server_key_iv: "738c519cd4fbf1ee798527aa9406bbfc"
server_key_version: "encryptionkey@0"
I20260926 20:56:12.807410 4675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "20e2e28b57024250838a2987a79857cc"
format_stamp: "Formatted at 2026-09-26 20:56:12 on dist-test-slave-fl7g"
server_key: "20c38bbd5962f42e5b773bd2a28f5868"
server_key_iv: "738c519cd4fbf1ee798527aa9406bbfc"
server_key_version: "encryptionkey@0"
I20260926 20:56:12.808822 4675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:56:12.809638 4689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:12.809909 4675 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:12.810060 4675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "20e2e28b57024250838a2987a79857cc"
format_stamp: "Formatted at 2026-09-26 20:56:12 on dist-test-slave-fl7g"
server_key: "20c38bbd5962f42e5b773bd2a28f5868"
server_key_iv: "738c519cd4fbf1ee798527aa9406bbfc"
server_key_version: "encryptionkey@0"
I20260926 20:56:12.810148 4675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:12.827745 4675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:12.830107 4675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:12.830322 4675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:12.830596 4675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:12.830967 4675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:12.831028 4675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:12.831074 4675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:12.831106 4675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:12.837718 4675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38875
I20260926 20:56:12.837757 4802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38875 every 8 connection(s)
I20260926 20:56:12.838066 4675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:56:12.843698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 4675
I20260926 20:56:12.843811 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:56:12.844014 392 external_mini_cluster.cc:1428] Setting key 0ae9a1977348de04715d11f888a57242
Sep 26 20:56:12 dist-test-slave-fl7g krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:12.848968 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:12.838986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46637 (local address 127.0.98.62:41029)
0926 20:56:12.839079 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:12.839082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:12.839272 (+ 190us) server_negotiation.cc:407] Connection header received
0926 20:56:12.839835 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:12.839837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:12.839850 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:12.839905 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:12.840994 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:12.841496 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:12.842572 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:12.842777 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:12.842855 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:12.845436 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:12.845438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:12.845439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:12.845449 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:12.847382 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:12.847844 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:12.847845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:12.847846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:12.847917 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:12.848211 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:12.848212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:12.848212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:12.848344 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:12.848451 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:12.848757 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 20:56:12.848852 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:56:12.849180 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:12.838869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41029 (local address 127.0.98.2:46637)
0926 20:56:12.839113 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:12.839176 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:12.839721 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:12.839994 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:12.839998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:12.840291 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:12.840885 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:12.840891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:12.841603 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:12.841604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:12.842473 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:12.842476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:12.842874 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:12.842875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:12.843006 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:12.843642 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:12.843653 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:12.845324 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:12.847490 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:12.847499 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:12.847501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:12.847756 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:12.848001 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:12.848002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:12.848003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:12.848122 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:12.848437 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:12.848444 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:12.848607 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 20:56:12.848731 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 20:56:12.848913 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":111,"threads_started":1}
I20260926 20:56:12.849769 4803 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41029
I20260926 20:56:12.849874 4803 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:12.850085 4803 heartbeater.cc:511] Master 127.0.98.62:41029 requested a full tablet report, sending...
I20260926 20:56:12.850518 4480 ts_manager.cc:194] Registered new tserver with Master: 20e2e28b57024250838a2987a79857cc (127.0.98.2:38875)
I20260926 20:56:12.851065 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46637
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:56:12.872816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41029
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:12.959789 4808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:12.960006 4808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:12.960031 4808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:12.961295 4808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:12.961344 4808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:12.961372 4808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:56:12.962808 4808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:12.963061 4808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:12.963260 4808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:12.968616 4816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:12.968863 4808 server_base.cc:1034] running on GCE node
W20260926 20:56:12.968616 4813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:12.968673 4814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:12.969187 4808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:12.969440 4808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:12.970615 4808 hybrid_clock.cc:648] HybridClock initialized: now 1790456172970605 us; error 30 us; skew 500 ppm
Sep 26 20:56:12 dist-test-slave-fl7g krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:12.973338 4808 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:56:12.973697 4808 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:12.973770 4808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:12.973925 4808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:12.975061 4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "b79a2f92e87541479475e281ea1e91f4"
format_stamp: "Formatted at 2026-09-26 20:56:12 on dist-test-slave-fl7g"
server_key: "1a5f28ea892f9bd8e91e976f67062332"
server_key_iv: "24cb63db6eb05e27619f2aeb129a43ec"
server_key_version: "encryptionkey@0"
I20260926 20:56:12.975463 4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b79a2f92e87541479475e281ea1e91f4"
format_stamp: "Formatted at 2026-09-26 20:56:12 on dist-test-slave-fl7g"
server_key: "1a5f28ea892f9bd8e91e976f67062332"
server_key_iv: "24cb63db6eb05e27619f2aeb129a43ec"
server_key_version: "encryptionkey@0"
I20260926 20:56:12.976888 4808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:56:12.977778 4822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:12.978020 4808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:12.978147 4808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "b79a2f92e87541479475e281ea1e91f4"
format_stamp: "Formatted at 2026-09-26 20:56:12 on dist-test-slave-fl7g"
server_key: "1a5f28ea892f9bd8e91e976f67062332"
server_key_iv: "24cb63db6eb05e27619f2aeb129a43ec"
server_key_version: "encryptionkey@0"
I20260926 20:56:12.978231 4808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:13.012812 4808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:13.015308 4808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:13.015522 4808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:13.015774 4808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:13.016101 4808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:13.016160 4808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:13.016218 4808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:13.016242 4808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:13.022760 4808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37059
I20260926 20:56:13.022818 4935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37059 every 8 connection(s)
I20260926 20:56:13.023154 4808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:56:13.028592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 4808
I20260926 20:56:13.028688 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:56:13.028915 392 external_mini_cluster.cc:1428] Setting key 307502c0a305b1f2c334bd454d2c0918
Sep 26 20:56:13 dist-test-slave-fl7g krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:13.034564 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.024039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50263 (local address 127.0.98.62:41029)
0926 20:56:13.024107 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.024109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.024378 (+ 269us) server_negotiation.cc:407] Connection header received
0926 20:56:13.025012 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.025014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.025028 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.025127 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:13.026159 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.026957 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.028136 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.028363 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.028475 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.031005 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:13.031007 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:13.031007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:13.031016 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:13.032844 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:13.033363 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:13.033364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:13.033365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:13.033444 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:13.033774 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:13.033775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:13.033775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:13.033946 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:13.034032 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:13.034337 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 20:56:13.034461 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:56:13.034765 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.024044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41029 (local address 127.0.98.3:50263)
0926 20:56:13.024248 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:13.024288 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:13.024884 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:13.025186 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:13.025189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:13.025467 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:13.026046 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.026052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.027084 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.027085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.028025 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.028028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.028548 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.028549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.028708 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.029426 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:13.029436 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:13.030891 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:13.032964 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:13.032975 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:13.032977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:13.033242 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:13.033543 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:13.033544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:13.033544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:13.033673 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:13.034026 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:13.034033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:13.034193 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 20:56:13.034316 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 20:56:13.034497 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260926 20:56:13.035557 4936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41029
I20260926 20:56:13.035688 4936 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:13.035869 4936 heartbeater.cc:511] Master 127.0.98.62:41029 requested a full tablet report, sending...
I20260926 20:56:13.036312 4480 ts_manager.cc:194] Registered new tserver with Master: b79a2f92e87541479475e281ea1e91f4 (127.0.98.3:37059)
I20260926 20:56:13.036736 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50263
I20260926 20:56:13.041085 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:56:13.049253 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.042709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42122 (local address 127.0.98.62:41029)
0926 20:56:13.042769 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.042772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.042872 (+ 100us) server_negotiation.cc:407] Connection header received
0926 20:56:13.042945 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.042946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.042957 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.043018 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:13.043616 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.044123 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.044952 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.045156 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.045245 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.046502 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:13.046503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:13.046504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:13.046512 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:13.047947 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:13.048344 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:13.048345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:13.048346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:13.048453 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:13.048659 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:13.048660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:13.048661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:13.048827 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:13.048937 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:13.049062 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 20:56:13.049146 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 20:56:13.051102 4480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:56:13.053066 4480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:56:13.060437 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.055973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56778 (local address 127.0.98.3:37059)
0926 20:56:13.056151 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.056156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.056168 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:56:13.056231 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.056233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.056252 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.056385 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:13.056576 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.057987 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.059261 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.060019 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.060142 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.060210 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:13.060295 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 20:56:13.060335 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
I20260926 20:56:13.060468 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.055554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38875 (local address 127.0.0.1:35032)
0926 20:56:13.055739 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:13.055785 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:13.055857 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:13.056593 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:13.056593 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:13.056595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:13.056674 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.056689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.058164 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.058165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.059256 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.059258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.060128 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.060129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.060202 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.060206 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:56:13.060238 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 20:56:13.060286 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260926 20:56:13.060535 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.056003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.0.98.1:39607)
0926 20:56:13.056162 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.056167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.056283 (+ 116us) server_negotiation.cc:407] Connection header received
0926 20:56:13.056354 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.056356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.056376 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.056482 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:13.056680 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.057545 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.059087 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.059780 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.059889 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.059968 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:13.060368 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 20:56:13.060442 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260926 20:56:13.060662 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.055750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.0.98.2:38875)
0926 20:56:13.056220 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.056224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.056246 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:56:13.056288 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.056290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.056305 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.056439 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:13.056753 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.058033 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.059331 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.060027 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.060364 (+ 337us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.060462 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:13.060490 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 20:56:13.060541 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":100,"threads_started":1}
I20260926 20:56:13.060731 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.055593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37059 (local address 127.0.0.1:56778)
0926 20:56:13.055740 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:13.055785 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:13.055857 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:13.056389 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:13.056393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:13.056395 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:13.056510 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.056513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.058105 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.058106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.059174 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.059176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.060149 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.060150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.060210 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.060214 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:56:13.060280 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 20:56:13.060329 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260926 20:56:13.060316 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.055788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39607 (local address 127.0.0.1:44914)
0926 20:56:13.056209 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:13.056219 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:13.056280 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:13.056541 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:13.056542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:13.056544 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:13.056595 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.056597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.057668 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.057669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.058988 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.058991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.059893 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.059894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.059983 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.060000 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 20:56:13.060136 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 20:56:13.060200 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":73,"threads_started":1}
I20260926 20:56:13.061764 4604 tablet_service.cc:1467] Processing CreateTablet for tablet 45fb989a46f04f6899beef6b2a7b69d6 (DEFAULT_TABLE table=test-table [id=a9ddf429c2e64d049e5e386033abe076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:13.061754 4870 tablet_service.cc:1467] Processing CreateTablet for tablet 45fb989a46f04f6899beef6b2a7b69d6 (DEFAULT_TABLE table=test-table [id=a9ddf429c2e64d049e5e386033abe076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:13.061844 4737 tablet_service.cc:1467] Processing CreateTablet for tablet 45fb989a46f04f6899beef6b2a7b69d6 (DEFAULT_TABLE table=test-table [id=a9ddf429c2e64d049e5e386033abe076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:13.062156 4604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45fb989a46f04f6899beef6b2a7b69d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:13.062150 4737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45fb989a46f04f6899beef6b2a7b69d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:13.062150 4870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45fb989a46f04f6899beef6b2a7b69d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:13.065578 4953 tablet_bootstrap.cc:492] T 45fb989a46f04f6899beef6b2a7b69d6 P 2eca83aa4c704116af6e0db1d2d8bf59: Bootstrap starting.
I20260926 20:56:13.065917 4955 tablet_bootstrap.cc:492] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4: Bootstrap starting.
I20260926 20:56:13.066372 4953 tablet_bootstrap.cc:654] T 45fb989a46f04f6899beef6b2a7b69d6 P 2eca83aa4c704116af6e0db1d2d8bf59: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:13.066531 4955 tablet_bootstrap.cc:654] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:13.066965 4953 log.cc:824] T 45fb989a46f04f6899beef6b2a7b69d6 P 2eca83aa4c704116af6e0db1d2d8bf59: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:13.067040 4955 log.cc:824] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:13.067415 4954 tablet_bootstrap.cc:492] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc: Bootstrap starting.
I20260926 20:56:13.067800 4953 tablet_bootstrap.cc:492] T 45fb989a46f04f6899beef6b2a7b69d6 P 2eca83aa4c704116af6e0db1d2d8bf59: No bootstrap required, opened a new log
I20260926 20:56:13.067852 4955 tablet_bootstrap.cc:492] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4: No bootstrap required, opened a new log
I20260926 20:56:13.067895 4953 ts_tablet_manager.cc:1397] T 45fb989a46f04f6899beef6b2a7b69d6 P 2eca83aa4c704116af6e0db1d2d8bf59: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:56:13.067909 4955 ts_tablet_manager.cc:1397] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:56:13.068104 4954 tablet_bootstrap.cc:654] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:13.068732 4954 log.cc:824] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:13.069691 4954 tablet_bootstrap.cc:492] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc: No bootstrap required, opened a new log
I20260926 20:56:13.069821 4954 ts_tablet_manager.cc:1397] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 20:56:13.070122 4955 raft_consensus.cc:348] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e2e28b57024250838a2987a79857cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } } peers { permanent_uuid: "b79a2f92e87541479475e281ea1e91f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37059 } } peers { permanent_uuid: "2eca83aa4c704116af6e0db1d2d8bf59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39607 } }
I20260926 20:56:13.070319 4955 raft_consensus.cc:374] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:13.070354 4955 raft_consensus.cc:729] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79a2f92e87541479475e281ea1e91f4, State: Initialized, Role: FOLLOWER
I20260926 20:56:13.070520 4955 consensus_queue.cc:260] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e2e28b57024250838a2987a79857cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } } peers { permanent_uuid: "b79a2f92e87541479475e281ea1e91f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37059 } } peers { permanent_uuid: "2eca83aa4c704116af6e0db1d2d8bf59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39607 } }
I20260926 20:56:13.070375 4953 raft_consensus.cc:348] T 45fb989a46f04f6899beef6b2a7b69d6 P 2eca83aa4c704116af6e0db1d2d8bf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e2e28b57024250838a2987a79857cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } } peers { permanent_uuid: "b79a2f92e87541479475e281ea1e91f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37059 } } peers { permanent_uuid: "2eca83aa4c704116af6e0db1d2d8bf59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39607 } }
I20260926 20:56:13.070629 4953 raft_consensus.cc:374] T 45fb989a46f04f6899beef6b2a7b69d6 P 2eca83aa4c704116af6e0db1d2d8bf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:13.070701 4953 raft_consensus.cc:729] T 45fb989a46f04f6899beef6b2a7b69d6 P 2eca83aa4c704116af6e0db1d2d8bf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eca83aa4c704116af6e0db1d2d8bf59, State: Initialized, Role: FOLLOWER
I20260926 20:56:13.070838 4955 ts_tablet_manager.cc:1428] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:56:13.070848 4953 consensus_queue.cc:260] T 45fb989a46f04f6899beef6b2a7b69d6 P 2eca83aa4c704116af6e0db1d2d8bf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e2e28b57024250838a2987a79857cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } } peers { permanent_uuid: "b79a2f92e87541479475e281ea1e91f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37059 } } peers { permanent_uuid: "2eca83aa4c704116af6e0db1d2d8bf59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39607 } }
I20260926 20:56:13.070986 4936 heartbeater.cc:503] Master 127.0.98.62:41029 was elected leader, sending a full tablet report...
I20260926 20:56:13.071246 4953 ts_tablet_manager.cc:1428] T 45fb989a46f04f6899beef6b2a7b69d6 P 2eca83aa4c704116af6e0db1d2d8bf59: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:56:13.071270 4670 heartbeater.cc:503] Master 127.0.98.62:41029 was elected leader, sending a full tablet report...
I20260926 20:56:13.071947 4954 raft_consensus.cc:348] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e2e28b57024250838a2987a79857cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } } peers { permanent_uuid: "b79a2f92e87541479475e281ea1e91f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37059 } } peers { permanent_uuid: "2eca83aa4c704116af6e0db1d2d8bf59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39607 } }
I20260926 20:56:13.072129 4954 raft_consensus.cc:374] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:13.072206 4954 raft_consensus.cc:729] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e2e28b57024250838a2987a79857cc, State: Initialized, Role: FOLLOWER
I20260926 20:56:13.072453 4954 consensus_queue.cc:260] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e2e28b57024250838a2987a79857cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } } peers { permanent_uuid: "b79a2f92e87541479475e281ea1e91f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37059 } } peers { permanent_uuid: "2eca83aa4c704116af6e0db1d2d8bf59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39607 } }
I20260926 20:56:13.072765 4954 ts_tablet_manager.cc:1428] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:56:13.072851 4803 heartbeater.cc:503] Master 127.0.98.62:41029 was elected leader, sending a full tablet report...
W20260926 20:56:13.088714 4804 tablet.cc:2369] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:56:13.177114 4671 tablet.cc:2369] T 45fb989a46f04f6899beef6b2a7b69d6 P 2eca83aa4c704116af6e0db1d2d8bf59: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:56:13.273715 4937 tablet.cc:2369] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:56:13.478904 4961 raft_consensus.cc:482] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:56:13.479053 4961 raft_consensus.cc:504] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e2e28b57024250838a2987a79857cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } } peers { permanent_uuid: "b79a2f92e87541479475e281ea1e91f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37059 } } peers { permanent_uuid: "2eca83aa4c704116af6e0db1d2d8bf59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39607 } }
I20260926 20:56:13.479420 4961 leader_election.cc:288] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b79a2f92e87541479475e281ea1e91f4 (127.0.98.3:37059), 2eca83aa4c704116af6e0db1d2d8bf59 (127.0.98.1:39607)
I20260926 20:56:13.482993 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.479652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37059 (local address 127.0.98.2:35503)
0926 20:56:13.479777 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:13.479790 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:13.479838 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:13.480043 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:13.480044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:13.480046 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:13.480111 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.480113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.480899 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.480901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.481999 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.482001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.482731 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.482731 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.482778 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.482782 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:56:13.482860 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:56:13.482902 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260926 20:56:13.482971 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.479747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35503 (local address 127.0.98.3:37059)
0926 20:56:13.479813 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.479816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.479867 (+ 51us) server_negotiation.cc:407] Connection header received
0926 20:56:13.479903 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.479904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.479915 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.479973 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:13.480197 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.480792 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.482087 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.482644 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.482717 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.482728 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:13.482866 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 20:56:13.482904 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:56:13.483417 4890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45fb989a46f04f6899beef6b2a7b69d6" candidate_uuid: "20e2e28b57024250838a2987a79857cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79a2f92e87541479475e281ea1e91f4" is_pre_election: true
I20260926 20:56:13.483629 4890 raft_consensus.cc:2454] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20e2e28b57024250838a2987a79857cc in term 0.
I20260926 20:56:13.483669 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.480003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38297 (local address 127.0.98.1:39607)
0926 20:56:13.480059 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.480061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.480070 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:56:13.480096 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.480097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.480107 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.480188 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:13.481028 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.481487 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.482819 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.483345 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.483417 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.483428 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:13.483561 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 20:56:13.483599 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:56:13.483669 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.479842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39607 (local address 127.0.98.2:38297)
0926 20:56:13.479980 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:13.479991 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:13.480034 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:13.480854 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:13.480855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:13.480858 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:13.480941 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.480942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.481574 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.481575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.482755 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.482757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.483425 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.483426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.483480 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.483484 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:56:13.483533 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 20:56:13.483583 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
I20260926 20:56:13.483907 4691 leader_election.cc:302] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20e2e28b57024250838a2987a79857cc, b79a2f92e87541479475e281ea1e91f4; no voters:
I20260926 20:56:13.484071 4961 raft_consensus.cc:2781] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:56:13.484050 4624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45fb989a46f04f6899beef6b2a7b69d6" candidate_uuid: "20e2e28b57024250838a2987a79857cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eca83aa4c704116af6e0db1d2d8bf59" is_pre_election: true
I20260926 20:56:13.484144 4961 raft_consensus.cc:482] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:56:13.484171 4961 raft_consensus.cc:3037] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:13.484208 4624 raft_consensus.cc:2454] T 45fb989a46f04f6899beef6b2a7b69d6 P 2eca83aa4c704116af6e0db1d2d8bf59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20e2e28b57024250838a2987a79857cc in term 0.
I20260926 20:56:13.484962 4961 raft_consensus.cc:504] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e2e28b57024250838a2987a79857cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } } peers { permanent_uuid: "b79a2f92e87541479475e281ea1e91f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37059 } } peers { permanent_uuid: "2eca83aa4c704116af6e0db1d2d8bf59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39607 } }
I20260926 20:56:13.485106 4961 leader_election.cc:288] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [CANDIDATE]: Term 1 election: Requested vote from peers b79a2f92e87541479475e281ea1e91f4 (127.0.98.3:37059), 2eca83aa4c704116af6e0db1d2d8bf59 (127.0.98.1:39607)
I20260926 20:56:13.485247 4890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45fb989a46f04f6899beef6b2a7b69d6" candidate_uuid: "20e2e28b57024250838a2987a79857cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79a2f92e87541479475e281ea1e91f4"
I20260926 20:56:13.485327 4890 raft_consensus.cc:3037] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:13.485333 4624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45fb989a46f04f6899beef6b2a7b69d6" candidate_uuid: "20e2e28b57024250838a2987a79857cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eca83aa4c704116af6e0db1d2d8bf59"
I20260926 20:56:13.485441 4624 raft_consensus.cc:3037] T 45fb989a46f04f6899beef6b2a7b69d6 P 2eca83aa4c704116af6e0db1d2d8bf59 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:13.486063 4890 raft_consensus.cc:2454] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20e2e28b57024250838a2987a79857cc in term 1.
I20260926 20:56:13.486212 4624 raft_consensus.cc:2454] T 45fb989a46f04f6899beef6b2a7b69d6 P 2eca83aa4c704116af6e0db1d2d8bf59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20e2e28b57024250838a2987a79857cc in term 1.
I20260926 20:56:13.486244 4691 leader_election.cc:302] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20e2e28b57024250838a2987a79857cc, b79a2f92e87541479475e281ea1e91f4; no voters:
I20260926 20:56:13.486326 4961 raft_consensus.cc:2781] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:13.486519 4961 raft_consensus.cc:686] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [term 1 LEADER]: Becoming Leader. State: Replica: 20e2e28b57024250838a2987a79857cc, State: Running, Role: LEADER
I20260926 20:56:13.486621 4961 consensus_queue.cc:237] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e2e28b57024250838a2987a79857cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } } peers { permanent_uuid: "b79a2f92e87541479475e281ea1e91f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37059 } } peers { permanent_uuid: "2eca83aa4c704116af6e0db1d2d8bf59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39607 } }
I20260926 20:56:13.487422 4480 catalog_manager.cc:5495] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 20e2e28b57024250838a2987a79857cc (127.0.98.2). New cstate: current_term: 1 leader_uuid: "20e2e28b57024250838a2987a79857cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e2e28b57024250838a2987a79857cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b79a2f92e87541479475e281ea1e91f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2eca83aa4c704116af6e0db1d2d8bf59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39607 } health_report { overall_health: UNKNOWN } } }
I20260926 20:56:13.507529 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.504943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.0.98.2:38875)
0926 20:56:13.505014 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.505016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.505027 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:56:13.505111 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.505112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.505135 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.505213 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:13.505378 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.505885 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.506686 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.506889 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.506967 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.507173 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:13.507283 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:13.507383 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:13.507401 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 20:56:13.507422 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:56:13.509743 4890 raft_consensus.cc:1261] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4 [term 1 FOLLOWER]: Refusing update from remote peer 20e2e28b57024250838a2987a79857cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:13.509794 4624 raft_consensus.cc:1261] T 45fb989a46f04f6899beef6b2a7b69d6 P 2eca83aa4c704116af6e0db1d2d8bf59 [term 1 FOLLOWER]: Refusing update from remote peer 20e2e28b57024250838a2987a79857cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:13.510181 4961 consensus_queue.cc:1035] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b79a2f92e87541479475e281ea1e91f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:13.510349 4964 consensus_queue.cc:1035] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eca83aa4c704116af6e0db1d2d8bf59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:13.512946 4967 mvcc.cc:205] Tried to move back new op lower bound from 7333708486692589568 to 7333708486601789440. Current Snapshot: MvccSnapshot[applied={T|T < 7333708486692589568}]
I20260926 20:56:13.513288 4969 mvcc.cc:205] Tried to move back new op lower bound from 7333708486692589568 to 7333708486601789440. Current Snapshot: MvccSnapshot[applied={T|T < 7333708486692589568}]
I20260926 20:56:13.516822 4480 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:13.516948 4480 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:13.517762 4480 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3da1a0d7e7b541ac909a783a7ad5181f: Sending DeleteTablet for 3 replicas of tablet 45fb989a46f04f6899beef6b2a7b69d6
I20260926 20:56:13.518082 4737 tablet_service.cc:1514] Processing DeleteTablet for tablet 45fb989a46f04f6899beef6b2a7b69d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35032
I20260926 20:56:13.518200 4870 tablet_service.cc:1514] Processing DeleteTablet for tablet 45fb989a46f04f6899beef6b2a7b69d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56778
I20260926 20:56:13.518229 4604 tablet_service.cc:1514] Processing DeleteTablet for tablet 45fb989a46f04f6899beef6b2a7b69d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44914
I20260926 20:56:13.518415 4978 tablet_replica.cc:330] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4: stopping tablet replica
I20260926 20:56:13.518481 4977 tablet_replica.cc:330] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc: stopping tablet replica
I20260926 20:56:13.518602 4977 raft_consensus.cc:2229] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:56:13.518599 4978 raft_consensus.cc:2229] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:13.518718 4978 raft_consensus.cc:2258] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:13.518785 4977 raft_consensus.cc:2258] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:13.518807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 4542
I20260926 20:56:13.519245 4977 ts_tablet_manager.cc:1905] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:13.519281 4978 ts_tablet_manager.cc:1905] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:13.521142 4978 ts_tablet_manager.cc:1918] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:13.521210 4978 log.cc:1196] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/45fb989a46f04f6899beef6b2a7b69d6
I20260926 20:56:13.521351 4977 ts_tablet_manager.cc:1918] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:13.521412 4977 log.cc:1196] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/45fb989a46f04f6899beef6b2a7b69d6
I20260926 20:56:13.521495 4978 ts_tablet_manager.cc:1939] T 45fb989a46f04f6899beef6b2a7b69d6 P b79a2f92e87541479475e281ea1e91f4: Deleting consensus metadata
I20260926 20:56:13.521659 4977 ts_tablet_manager.cc:1939] T 45fb989a46f04f6899beef6b2a7b69d6 P 20e2e28b57024250838a2987a79857cc: Deleting consensus metadata
I20260926 20:56:13.522094 4467 catalog_manager.cc:4841] TS 20e2e28b57024250838a2987a79857cc (127.0.98.2:38875): tablet 45fb989a46f04f6899beef6b2a7b69d6 (table test-table [id=a9ddf429c2e64d049e5e386033abe076]) successfully deleted
I20260926 20:56:13.522855 4465 catalog_manager.cc:4841] TS b79a2f92e87541479475e281ea1e91f4 (127.0.98.3:37059): tablet 45fb989a46f04f6899beef6b2a7b69d6 (table test-table [id=a9ddf429c2e64d049e5e386033abe076]) successfully deleted
W20260926 20:56:13.527066 4465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39607 (error 108)
I20260926 20:56:13.527172 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 4675
I20260926 20:56:13.528386 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.528143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39607 (local address 127.0.0.1:44918)
0926 20:56:13.528317 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:13.528355 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260926 20:56:13.528952 4465 catalog_manager.cc:4584] TS 2eca83aa4c704116af6e0db1d2d8bf59 (127.0.98.1:39607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 45fb989a46f04f6899beef6b2a7b69d6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39607: connect: Connection refused (error 111)
I20260926 20:56:13.532050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 4808
I20260926 20:56:13.536561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 4449
2026-09-26T20:56:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2345 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T20:56:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:13Z Disabled control of system clock
I20260926 20:56:13.564347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38235
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:56:13.646224 4984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:13.646468 4984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:13.646492 4984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:13.647640 4984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:13.647687 4984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:13.647701 4984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:13.647713 4984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:13.647727 4984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:13.649204 4984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38235
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:13.649417 4984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:13.649684 4984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:13.654757 4990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:13.654714 4992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:13.654723 4989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:13.654927 4984 server_base.cc:1034] running on GCE node
I20260926 20:56:13.655192 4984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:13.655476 4984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:13.656613 4984 hybrid_clock.cc:648] HybridClock initialized: now 1790456173656602 us; error 32 us; skew 500 ppm
I20260926 20:56:13.657902 4984 webserver.cc:460] Webserver started at http://127.0.98.62:38293/ using document root <none> and password file <none>
I20260926 20:56:13.658174 4984 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:13.658223 4984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:13.658360 4984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:13.659515 4984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "4ae5e41e94364a8d8d0376aba1416237"
format_stamp: "Formatted at 2026-09-26 20:56:13 on dist-test-slave-fl7g"
server_key: "97d67ef43f46b05816e4ca4c4e78a194"
server_key_iv: "bd64c070daa8f603aeac4c3a571365f3"
server_key_version: "encryptionkey@0"
I20260926 20:56:13.659888 4984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "4ae5e41e94364a8d8d0376aba1416237"
format_stamp: "Formatted at 2026-09-26 20:56:13 on dist-test-slave-fl7g"
server_key: "97d67ef43f46b05816e4ca4c4e78a194"
server_key_iv: "bd64c070daa8f603aeac4c3a571365f3"
server_key_version: "encryptionkey@0"
I20260926 20:56:13.661340 4984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:56:13.662194 4998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:13.662575 4984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:13.662659 4984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "4ae5e41e94364a8d8d0376aba1416237"
format_stamp: "Formatted at 2026-09-26 20:56:13 on dist-test-slave-fl7g"
server_key: "97d67ef43f46b05816e4ca4c4e78a194"
server_key_iv: "bd64c070daa8f603aeac4c3a571365f3"
server_key_version: "encryptionkey@0"
I20260926 20:56:13.662760 4984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:13.666281 4984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:13.666568 4984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:13.666698 4984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:13.670578 4984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38235
I20260926 20:56:13.670631 5050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38235 every 8 connection(s)
I20260926 20:56:13.670981 4984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:13.671681 5051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:13.674510 5051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237: Bootstrap starting.
I20260926 20:56:13.675071 5051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:13.675310 5051 log.cc:824] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:13.675974 5051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237: No bootstrap required, opened a new log
I20260926 20:56:13.677646 5051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae5e41e94364a8d8d0376aba1416237" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38235 } }
I20260926 20:56:13.677801 5051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:13.677825 5051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae5e41e94364a8d8d0376aba1416237, State: Initialized, Role: FOLLOWER
I20260926 20:56:13.677934 5051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae5e41e94364a8d8d0376aba1416237" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38235 } }
I20260926 20:56:13.678014 5051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:13.678051 5051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:13.678126 5051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:13.678692 5051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae5e41e94364a8d8d0376aba1416237" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38235 } }
I20260926 20:56:13.678814 5051 leader_election.cc:302] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ae5e41e94364a8d8d0376aba1416237; no voters:
I20260926 20:56:13.678877 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 4984
I20260926 20:56:13.679014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:56:13.679039 5051 leader_election.cc:288] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:13.679078 5054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:13.679219 5054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237 [term 1 LEADER]: Becoming Leader. State: Replica: 4ae5e41e94364a8d8d0376aba1416237, State: Running, Role: LEADER
I20260926 20:56:13.679221 392 external_mini_cluster.cc:1428] Setting key bdfc54de156c9a723ccee06664528bbe
I20260926 20:56:13.679309 5051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:13.679298 5054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae5e41e94364a8d8d0376aba1416237" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38235 } }
I20260926 20:56:13.679603 5054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ae5e41e94364a8d8d0376aba1416237. Latest consensus state: current_term: 1 leader_uuid: "4ae5e41e94364a8d8d0376aba1416237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae5e41e94364a8d8d0376aba1416237" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38235 } } }
I20260926 20:56:13.679669 5054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:13.680153 5060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:13.680315 5056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ae5e41e94364a8d8d0376aba1416237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae5e41e94364a8d8d0376aba1416237" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38235 } } }
I20260926 20:56:13.680503 5056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:13.680725 5060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:13.682564 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.679951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37516 (local address 127.0.98.62:38235)
0926 20:56:13.680902 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.680911 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.680943 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:56:13.681058 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.681065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.681110 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.681267 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:13.682004 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:13.682008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:13.682008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:13.682013 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:13.682058 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:13.682074 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:13.682159 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:13.682265 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 20:56:13.682340 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":91,"threads_started":1}
I20260926 20:56:13.683048 5060 catalog_manager.cc:1347] Generated new cluster ID: 6aa5f2f5de8a454f8daa2d64a47c472e
I20260926 20:56:13.683099 5060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:56:13.691748 5060 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:13.691977 5060 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:13.699174 5060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237: Generated new TSK 0
I20260926 20:56:13.699437 5060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:56:13.709268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38235
--builtin_ntp_servers=127.0.98.20:45043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:56:13.793664 5075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:13.793875 5075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:13.793900 5075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:13.795147 5075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:13.795195 5075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:13.795224 5075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:56:13.796666 5075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:13.796941 5075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:13.797271 5075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:13.803030 5080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:13.803202 5083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:13.803193 5081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:13.803169 5075 server_base.cc:1034] running on GCE node
I20260926 20:56:13.803689 5075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:13.804014 5075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:13.805183 5075 hybrid_clock.cc:648] HybridClock initialized: now 1790456173805171 us; error 28 us; skew 500 ppm
I20260926 20:56:13.806622 5075 webserver.cc:460] Webserver started at http://127.0.98.1:33697/ using document root <none> and password file <none>
I20260926 20:56:13.806912 5075 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:13.806959 5075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:13.807108 5075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:13.808267 5075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "0494b540724f4ae88f279161bbbd5c05"
format_stamp: "Formatted at 2026-09-26 20:56:13 on dist-test-slave-fl7g"
server_key: "0dff6f61701ec47d4edba85ac4aef401"
server_key_iv: "98971c19686663d4713993bed2615797"
server_key_version: "encryptionkey@0"
I20260926 20:56:13.808719 5075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0494b540724f4ae88f279161bbbd5c05"
format_stamp: "Formatted at 2026-09-26 20:56:13 on dist-test-slave-fl7g"
server_key: "0dff6f61701ec47d4edba85ac4aef401"
server_key_iv: "98971c19686663d4713993bed2615797"
server_key_version: "encryptionkey@0"
I20260926 20:56:13.810119 5075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:13.810866 5089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:13.811065 5075 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:13.811160 5075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "0494b540724f4ae88f279161bbbd5c05"
format_stamp: "Formatted at 2026-09-26 20:56:13 on dist-test-slave-fl7g"
server_key: "0dff6f61701ec47d4edba85ac4aef401"
server_key_iv: "98971c19686663d4713993bed2615797"
server_key_version: "encryptionkey@0"
I20260926 20:56:13.811244 5075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:13.814826 5075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:13.815110 5075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:13.815253 5075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:13.815460 5075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:13.815768 5075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:13.815824 5075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:13.815892 5075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:13.815922 5075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:13.821869 5075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46381
I20260926 20:56:13.821911 5202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46381 every 8 connection(s)
I20260926 20:56:13.822247 5075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:56:13.824100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 5075
I20260926 20:56:13.824232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:56:13.824499 392 external_mini_cluster.cc:1428] Setting key 27d5454b5a34ee5764f18270ee84de2b
I20260926 20:56:13.825501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38235
--builtin_ntp_servers=127.0.98.20:45043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 20:56:13.825801 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.823088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56379 (local address 127.0.98.62:38235)
0926 20:56:13.823165 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.823167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.823411 (+ 244us) server_negotiation.cc:407] Connection header received
0926 20:56:13.823955 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.823956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.823965 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.824008 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:13.825514 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:13.825515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:13.825516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:13.825520 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:13.825539 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:13.825545 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:13.825599 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:13.825715 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 20:56:13.825732 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:56:13.825999 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.823062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38235 (local address 127.0.98.1:56379)
0926 20:56:13.823281 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:13.823317 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:13.823833 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:13.824093 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:13.824096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:13.824104 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:13.825335 (+ 1231us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:56:13.825336 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:13.825376 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:56:13.825406 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:13.825630 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:13.825636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:13.825640 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:56:13.825713 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 20:56:13.825778 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260926 20:56:13.826455 5203 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38235
I20260926 20:56:13.826567 5203 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:13.826731 5203 heartbeater.cc:511] Master 127.0.98.62:38235 requested a full tablet report, sending...
I20260926 20:56:13.827342 5015 ts_manager.cc:194] Registered new tserver with Master: 0494b540724f4ae88f279161bbbd5c05 (127.0.98.1:46381)
W20260926 20:56:13.909962 5206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:13.910255 5206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:13.910288 5206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:13.911685 5206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:13.911734 5206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:13.911763 5206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:56:13.913223 5206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:13.913437 5206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:13.913717 5206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:13.919025 5211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:13.919075 5214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:13.919056 5212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:13.919327 5206 server_base.cc:1034] running on GCE node
I20260926 20:56:13.919564 5206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:13.919832 5206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:13.921025 5206 hybrid_clock.cc:648] HybridClock initialized: now 1790456173921021 us; error 45 us; skew 500 ppm
I20260926 20:56:13.922482 5206 webserver.cc:460] Webserver started at http://127.0.98.2:38973/ using document root <none> and password file <none>
I20260926 20:56:13.922749 5206 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:13.922824 5206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:13.922976 5206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:13.924094 5206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "721fcfa79c5847d1adae9d3dad5db772"
format_stamp: "Formatted at 2026-09-26 20:56:13 on dist-test-slave-fl7g"
server_key: "74a78ef3c6d65aeb4289c68a26ece350"
server_key_iv: "b65ebb95fd168f295e474bcb20f99368"
server_key_version: "encryptionkey@0"
I20260926 20:56:13.924537 5206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "721fcfa79c5847d1adae9d3dad5db772"
format_stamp: "Formatted at 2026-09-26 20:56:13 on dist-test-slave-fl7g"
server_key: "74a78ef3c6d65aeb4289c68a26ece350"
server_key_iv: "b65ebb95fd168f295e474bcb20f99368"
server_key_version: "encryptionkey@0"
I20260926 20:56:13.925967 5206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:13.926782 5220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:13.927002 5206 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:13.927105 5206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "721fcfa79c5847d1adae9d3dad5db772"
format_stamp: "Formatted at 2026-09-26 20:56:13 on dist-test-slave-fl7g"
server_key: "74a78ef3c6d65aeb4289c68a26ece350"
server_key_iv: "b65ebb95fd168f295e474bcb20f99368"
server_key_version: "encryptionkey@0"
I20260926 20:56:13.927193 5206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:13.930708 5206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:13.930990 5206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:13.931134 5206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:13.931357 5206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:13.931681 5206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:13.931737 5206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:13.931778 5206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:13.931804 5206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:13.937892 5206 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42033
I20260926 20:56:13.937933 5333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42033 every 8 connection(s)
I20260926 20:56:13.938289 5206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:56:13.941087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 5206
I20260926 20:56:13.941231 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:56:13.941452 392 external_mini_cluster.cc:1428] Setting key 5e8da4d9ecfc70c168a3eca00cc6c97a
I20260926 20:56:13.942055 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.939017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33481 (local address 127.0.98.62:38235)
0926 20:56:13.939066 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.939068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.939370 (+ 302us) server_negotiation.cc:407] Connection header received
0926 20:56:13.939925 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.939926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.939933 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.939983 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:13.941685 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:13.941687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:13.941688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:13.941694 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:13.941717 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:13.941723 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:13.941788 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:13.941967 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 20:56:13.941985 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:56:13.942306 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.939007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38235 (local address 127.0.98.2:33481)
0926 20:56:13.939224 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:13.939271 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:13.939805 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:13.940062 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:13.940067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:13.940077 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:13.941476 (+ 1399us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:56:13.941477 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:13.941523 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:56:13.941564 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:13.941861 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:13.941866 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:13.941869 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:56:13.941963 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:56:13.942049 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260926 20:56:13.942515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38235
--builtin_ntp_servers=127.0.98.20:45043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 20:56:13.942849 5334 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38235
I20260926 20:56:13.942966 5334 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:13.943125 5334 heartbeater.cc:511] Master 127.0.98.62:38235 requested a full tablet report, sending...
I20260926 20:56:13.943497 5015 ts_manager.cc:194] Registered new tserver with Master: 721fcfa79c5847d1adae9d3dad5db772 (127.0.98.2:42033)
W20260926 20:56:14.025012 5337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:14.025241 5337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:14.025265 5337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:14.026513 5337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:14.026561 5337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:14.026588 5337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:56:14.028000 5337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:14.028316 5337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:14.028697 5337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:14.034618 5342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:14.034649 5345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:14.034670 5337 server_base.cc:1034] running on GCE node
W20260926 20:56:14.034649 5343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:14.035147 5337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:14.035403 5337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:14.036708 5337 hybrid_clock.cc:648] HybridClock initialized: now 1790456174036675 us; error 54 us; skew 500 ppm
I20260926 20:56:14.038189 5337 webserver.cc:460] Webserver started at http://127.0.98.3:42899/ using document root <none> and password file <none>
I20260926 20:56:14.038475 5337 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:14.038565 5337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:14.038728 5337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:14.039927 5337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "73d71fc57b804e8b9b7d1f6e448002f1"
format_stamp: "Formatted at 2026-09-26 20:56:14 on dist-test-slave-fl7g"
server_key: "6b23d07665bdca87bca234bb7f828a78"
server_key_iv: "c039493a49369a0063ad7d3926a0158d"
server_key_version: "encryptionkey@0"
I20260926 20:56:14.040299 5337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "73d71fc57b804e8b9b7d1f6e448002f1"
format_stamp: "Formatted at 2026-09-26 20:56:14 on dist-test-slave-fl7g"
server_key: "6b23d07665bdca87bca234bb7f828a78"
server_key_iv: "c039493a49369a0063ad7d3926a0158d"
server_key_version: "encryptionkey@0"
I20260926 20:56:14.041718 5337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:14.042606 5351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:14.042785 5337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:14.042894 5337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "73d71fc57b804e8b9b7d1f6e448002f1"
format_stamp: "Formatted at 2026-09-26 20:56:14 on dist-test-slave-fl7g"
server_key: "6b23d07665bdca87bca234bb7f828a78"
server_key_iv: "c039493a49369a0063ad7d3926a0158d"
server_key_version: "encryptionkey@0"
I20260926 20:56:14.042972 5337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:14.046619 5337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:14.046924 5337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:14.047077 5337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:14.047303 5337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:14.047676 5337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:14.047737 5337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:14.047778 5337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:14.047804 5337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:14.053902 5337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42143
I20260926 20:56:14.053943 5464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42143 every 8 connection(s)
I20260926 20:56:14.054289 5337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:56:14.057765 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.055098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58303 (local address 127.0.98.62:38235)
0926 20:56:14.055168 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:14.055171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:14.055489 (+ 318us) server_negotiation.cc:407] Connection header received
0926 20:56:14.055988 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:14.055989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:14.055995 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:14.056060 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:14.057366 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:14.057368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:14.057368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:14.057373 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:14.057395 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:14.057400 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:14.057497 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:14.057644 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 20:56:14.057663 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:56:14.057819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 5337
I20260926 20:56:14.057878 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.055091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38235 (local address 127.0.98.3:58303)
0926 20:56:14.055360 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:14.055397 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:14.055863 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:14.056137 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:14.056141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:14.056150 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:14.057181 (+ 1031us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:56:14.057182 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:14.057213 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:56:14.057247 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:14.057496 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:14.057500 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:14.057502 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:56:14.057573 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:56:14.057656 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":99,"threads_started":1}
I20260926 20:56:14.057925 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:56:14.058141 392 external_mini_cluster.cc:1428] Setting key 4109fa5c4f97e0ad96881e9155a8a052
I20260926 20:56:14.058396 5465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38235
I20260926 20:56:14.058530 5465 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:14.058700 5465 heartbeater.cc:511] Master 127.0.98.62:38235 requested a full tablet report, sending...
I20260926 20:56:14.059116 5015 ts_manager.cc:194] Registered new tserver with Master: 73d71fc57b804e8b9b7d1f6e448002f1 (127.0.98.3:42143)
I20260926 20:56:14.070047 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:56:14.072764 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.071514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.0.98.62:38235)
0926 20:56:14.071586 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:14.071588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:14.071659 (+ 71us) server_negotiation.cc:407] Connection header received
0926 20:56:14.071723 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:14.071724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:14.071728 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:14.071778 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:14.072443 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:14.072444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:14.072445 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:14.072450 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:14.072492 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:14.072497 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:14.072562 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:14.072655 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 20:56:14.072673 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:56:14.074406 5015 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:56:14.075971 5015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:56:14.079931 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.078204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46381 (local address 127.0.0.1:33818)
0926 20:56:14.078485 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:14.078526 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:14.078627 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:14.078827 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:14.078831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:14.078838 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:14.079441 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:56:14.079441 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:14.079459 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:56:14.079486 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:14.079767 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:14.079772 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:14.079776 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:56:14.079842 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 20:56:14.079857 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":77,"threads_started":1}
I20260926 20:56:14.079946 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.078294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.0.98.1:46381)
0926 20:56:14.078477 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:14.078483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:14.078630 (+ 147us) server_negotiation.cc:407] Connection header received
0926 20:56:14.078713 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:14.078717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:14.078731 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:14.078816 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:14.079595 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:14.079603 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:14.079604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:14.079608 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:14.079645 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:14.079670 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:14.079748 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:14.079852 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 20:56:14.079867 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260926 20:56:14.080587 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.078241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42033 (local address 127.0.0.1:44442)
0926 20:56:14.078757 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:14.078766 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:14.078889 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:14.079458 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:14.079458 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:14.079461 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:14.079891 (+ 430us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:56:14.079892 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:14.079898 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:56:14.079910 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:14.080483 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:14.080483 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:14.080484 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:56:14.080533 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 20:56:14.080542 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":91,"threads_started":1}
I20260926 20:56:14.080765 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.078463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44442 (local address 127.0.98.2:42033)
0926 20:56:14.078641 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:14.078645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:14.078866 (+ 221us) server_negotiation.cc:407] Connection header received
0926 20:56:14.079092 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:14.079098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:14.079108 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:14.079181 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:14.080205 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:14.080211 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:14.080211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:14.080216 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:14.080245 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:14.080264 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:14.080297 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:14.080542 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 20:56:14.080561 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260926 20:56:14.080893 5137 tablet_service.cc:1467] Processing CreateTablet for tablet d437b34c420f4f13a8c43117797d04b1 (DEFAULT_TABLE table=test-table [id=a029ea04676a4317a0361c8139c7f034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:14.081314 5137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d437b34c420f4f13a8c43117797d04b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:14.081419 5268 tablet_service.cc:1467] Processing CreateTablet for tablet d437b34c420f4f13a8c43117797d04b1 (DEFAULT_TABLE table=test-table [id=a029ea04676a4317a0361c8139c7f034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:14.081748 5268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d437b34c420f4f13a8c43117797d04b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:14.084096 5479 tablet_bootstrap.cc:492] T d437b34c420f4f13a8c43117797d04b1 P 0494b540724f4ae88f279161bbbd5c05: Bootstrap starting.
I20260926 20:56:14.084339 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.078222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42143 (local address 127.0.0.1:57842)
0926 20:56:14.078636 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:14.078647 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:14.078938 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:14.079682 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:14.079683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:14.079685 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:14.080335 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:56:14.080335 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:14.080340 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:56:14.080349 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:14.084213 (+ 3864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:14.084214 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:14.084215 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:56:14.084259 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 20:56:14.084274 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":106,"threads_started":1}
I20260926 20:56:14.084385 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.078463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.0.98.3:42143)
0926 20:56:14.078755 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:14.078758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:14.078771 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:56:14.079044 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:14.079050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:14.079061 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:14.079128 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:14.084033 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:14.084043 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:14.084043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:14.084048 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:14.084090 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:14.084125 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:14.084187 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:14.084282 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 20:56:14.084299 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":87,"threads_started":1}
I20260926 20:56:14.084481 5480 tablet_bootstrap.cc:492] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772: Bootstrap starting.
I20260926 20:56:14.085012 5399 tablet_service.cc:1467] Processing CreateTablet for tablet d437b34c420f4f13a8c43117797d04b1 (DEFAULT_TABLE table=test-table [id=a029ea04676a4317a0361c8139c7f034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:14.085171 5480 tablet_bootstrap.cc:654] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:14.085281 5479 tablet_bootstrap.cc:654] T d437b34c420f4f13a8c43117797d04b1 P 0494b540724f4ae88f279161bbbd5c05: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:14.085371 5399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d437b34c420f4f13a8c43117797d04b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:14.085515 5480 log.cc:824] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:14.085577 5479 log.cc:824] T d437b34c420f4f13a8c43117797d04b1 P 0494b540724f4ae88f279161bbbd5c05: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:14.086244 5479 tablet_bootstrap.cc:492] T d437b34c420f4f13a8c43117797d04b1 P 0494b540724f4ae88f279161bbbd5c05: No bootstrap required, opened a new log
I20260926 20:56:14.086299 5479 ts_tablet_manager.cc:1397] T d437b34c420f4f13a8c43117797d04b1 P 0494b540724f4ae88f279161bbbd5c05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:56:14.088480 5483 tablet_bootstrap.cc:492] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1: Bootstrap starting.
I20260926 20:56:14.088455 5479 raft_consensus.cc:348] T d437b34c420f4f13a8c43117797d04b1 P 0494b540724f4ae88f279161bbbd5c05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d71fc57b804e8b9b7d1f6e448002f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42143 } } peers { permanent_uuid: "0494b540724f4ae88f279161bbbd5c05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } } peers { permanent_uuid: "721fcfa79c5847d1adae9d3dad5db772" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } }
I20260926 20:56:14.088640 5479 raft_consensus.cc:374] T d437b34c420f4f13a8c43117797d04b1 P 0494b540724f4ae88f279161bbbd5c05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:14.088685 5479 raft_consensus.cc:729] T d437b34c420f4f13a8c43117797d04b1 P 0494b540724f4ae88f279161bbbd5c05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0494b540724f4ae88f279161bbbd5c05, State: Initialized, Role: FOLLOWER
I20260926 20:56:14.088826 5479 consensus_queue.cc:260] T d437b34c420f4f13a8c43117797d04b1 P 0494b540724f4ae88f279161bbbd5c05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d71fc57b804e8b9b7d1f6e448002f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42143 } } peers { permanent_uuid: "0494b540724f4ae88f279161bbbd5c05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } } peers { permanent_uuid: "721fcfa79c5847d1adae9d3dad5db772" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } }
I20260926 20:56:14.089102 5479 ts_tablet_manager.cc:1428] T d437b34c420f4f13a8c43117797d04b1 P 0494b540724f4ae88f279161bbbd5c05: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:56:14.089129 5203 heartbeater.cc:503] Master 127.0.98.62:38235 was elected leader, sending a full tablet report...
I20260926 20:56:14.089174 5483 tablet_bootstrap.cc:654] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:14.089493 5483 log.cc:824] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:14.089596 5480 tablet_bootstrap.cc:492] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772: No bootstrap required, opened a new log
I20260926 20:56:14.089676 5480 ts_tablet_manager.cc:1397] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 20:56:14.090518 5483 tablet_bootstrap.cc:492] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1: No bootstrap required, opened a new log
I20260926 20:56:14.090590 5483 ts_tablet_manager.cc:1397] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:56:14.092463 5480 raft_consensus.cc:348] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d71fc57b804e8b9b7d1f6e448002f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42143 } } peers { permanent_uuid: "0494b540724f4ae88f279161bbbd5c05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } } peers { permanent_uuid: "721fcfa79c5847d1adae9d3dad5db772" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } }
I20260926 20:56:14.092676 5480 raft_consensus.cc:374] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:14.092720 5480 raft_consensus.cc:729] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721fcfa79c5847d1adae9d3dad5db772, State: Initialized, Role: FOLLOWER
I20260926 20:56:14.092864 5480 consensus_queue.cc:260] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d71fc57b804e8b9b7d1f6e448002f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42143 } } peers { permanent_uuid: "0494b540724f4ae88f279161bbbd5c05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } } peers { permanent_uuid: "721fcfa79c5847d1adae9d3dad5db772" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } }
I20260926 20:56:14.093166 5480 ts_tablet_manager.cc:1428] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 20:56:14.093194 5334 heartbeater.cc:503] Master 127.0.98.62:38235 was elected leader, sending a full tablet report...
I20260926 20:56:14.093274 5483 raft_consensus.cc:348] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d71fc57b804e8b9b7d1f6e448002f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42143 } } peers { permanent_uuid: "0494b540724f4ae88f279161bbbd5c05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } } peers { permanent_uuid: "721fcfa79c5847d1adae9d3dad5db772" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } }
I20260926 20:56:14.093449 5483 raft_consensus.cc:374] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:14.093488 5483 raft_consensus.cc:729] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d71fc57b804e8b9b7d1f6e448002f1, State: Initialized, Role: FOLLOWER
I20260926 20:56:14.093781 5483 consensus_queue.cc:260] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d71fc57b804e8b9b7d1f6e448002f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42143 } } peers { permanent_uuid: "0494b540724f4ae88f279161bbbd5c05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } } peers { permanent_uuid: "721fcfa79c5847d1adae9d3dad5db772" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } }
I20260926 20:56:14.094081 5483 ts_tablet_manager.cc:1428] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:56:14.094146 5465 heartbeater.cc:503] Master 127.0.98.62:38235 was elected leader, sending a full tablet report...
W20260926 20:56:14.188923 5335 tablet.cc:2369] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:56:14.304910 5466 tablet.cc:2369] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:56:14.305781 5486 raft_consensus.cc:482] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:56:14.305912 5486 raft_consensus.cc:504] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d71fc57b804e8b9b7d1f6e448002f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42143 } } peers { permanent_uuid: "0494b540724f4ae88f279161bbbd5c05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } } peers { permanent_uuid: "721fcfa79c5847d1adae9d3dad5db772" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } }
I20260926 20:56:14.306352 5486 leader_election.cc:288] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73d71fc57b804e8b9b7d1f6e448002f1 (127.0.98.3:42143), 0494b540724f4ae88f279161bbbd5c05 (127.0.98.1:46381)
I20260926 20:56:14.307868 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.306621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36959 (local address 127.0.98.3:42143)
0926 20:56:14.306663 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:14.306665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:14.306675 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:56:14.306704 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:14.306705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:14.306710 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:14.306778 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:14.307572 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:14.307573 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:14.307574 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:14.307578 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:14.307599 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:14.307609 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:14.307685 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:14.307772 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 20:56:14.307787 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:56:14.307875 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.306536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42143 (local address 127.0.98.2:36959)
0926 20:56:14.306590 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:14.306603 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:14.306647 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:14.306808 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:14.306809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:14.306812 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:14.307421 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:56:14.307422 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:14.307429 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:56:14.307441 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:14.307704 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:14.307705 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:14.307706 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:56:14.307761 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 20:56:14.307790 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 20:56:14.307940 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.306669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58671 (local address 127.0.98.1:46381)
0926 20:56:14.306710 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:14.306712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:14.306870 (+ 158us) server_negotiation.cc:407] Connection header received
0926 20:56:14.306905 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:14.306906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:14.306913 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:14.306987 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:14.307696 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:14.307697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:14.307698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:14.307702 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:14.307719 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:14.307723 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:14.307800 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:14.307856 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 20:56:14.307868 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:56:14.308005 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.306603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46381 (local address 127.0.98.2:58671)
0926 20:56:14.306781 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:14.306792 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:14.306847 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:14.307051 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:14.307052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:14.307070 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:14.307576 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:56:14.307577 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:14.307584 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:56:14.307603 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:14.307806 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:14.307807 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:14.307807 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 20:56:14.307857 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 20:56:14.307866 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":104,"threads_started":1}
I20260926 20:56:14.308285 5419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d437b34c420f4f13a8c43117797d04b1" candidate_uuid: "721fcfa79c5847d1adae9d3dad5db772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d71fc57b804e8b9b7d1f6e448002f1" is_pre_election: true
I20260926 20:56:14.308473 5419 raft_consensus.cc:2454] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721fcfa79c5847d1adae9d3dad5db772 in term 0.
I20260926 20:56:14.308459 5157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d437b34c420f4f13a8c43117797d04b1" candidate_uuid: "721fcfa79c5847d1adae9d3dad5db772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0494b540724f4ae88f279161bbbd5c05" is_pre_election: true
I20260926 20:56:14.308681 5157 raft_consensus.cc:2454] T d437b34c420f4f13a8c43117797d04b1 P 0494b540724f4ae88f279161bbbd5c05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721fcfa79c5847d1adae9d3dad5db772 in term 0.
I20260926 20:56:14.308754 5224 leader_election.cc:302] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721fcfa79c5847d1adae9d3dad5db772, 73d71fc57b804e8b9b7d1f6e448002f1; no voters:
I20260926 20:56:14.308894 5486 raft_consensus.cc:2781] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:56:14.308974 5486 raft_consensus.cc:482] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:56:14.309010 5486 raft_consensus.cc:3037] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:14.309721 5486 raft_consensus.cc:504] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d71fc57b804e8b9b7d1f6e448002f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42143 } } peers { permanent_uuid: "0494b540724f4ae88f279161bbbd5c05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } } peers { permanent_uuid: "721fcfa79c5847d1adae9d3dad5db772" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } }
I20260926 20:56:14.309880 5486 leader_election.cc:288] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [CANDIDATE]: Term 1 election: Requested vote from peers 73d71fc57b804e8b9b7d1f6e448002f1 (127.0.98.3:42143), 0494b540724f4ae88f279161bbbd5c05 (127.0.98.1:46381)
I20260926 20:56:14.310010 5419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d437b34c420f4f13a8c43117797d04b1" candidate_uuid: "721fcfa79c5847d1adae9d3dad5db772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d71fc57b804e8b9b7d1f6e448002f1"
I20260926 20:56:14.310060 5157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d437b34c420f4f13a8c43117797d04b1" candidate_uuid: "721fcfa79c5847d1adae9d3dad5db772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0494b540724f4ae88f279161bbbd5c05"
I20260926 20:56:14.310096 5419 raft_consensus.cc:3037] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:14.310129 5157 raft_consensus.cc:3037] T d437b34c420f4f13a8c43117797d04b1 P 0494b540724f4ae88f279161bbbd5c05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:14.311024 5157 raft_consensus.cc:2454] T d437b34c420f4f13a8c43117797d04b1 P 0494b540724f4ae88f279161bbbd5c05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721fcfa79c5847d1adae9d3dad5db772 in term 1.
I20260926 20:56:14.311025 5419 raft_consensus.cc:2454] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721fcfa79c5847d1adae9d3dad5db772 in term 1.
I20260926 20:56:14.311259 5222 leader_election.cc:302] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0494b540724f4ae88f279161bbbd5c05, 721fcfa79c5847d1adae9d3dad5db772; no voters:
I20260926 20:56:14.311362 5486 raft_consensus.cc:2781] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:14.311529 5486 raft_consensus.cc:686] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [term 1 LEADER]: Becoming Leader. State: Replica: 721fcfa79c5847d1adae9d3dad5db772, State: Running, Role: LEADER
I20260926 20:56:14.311646 5486 consensus_queue.cc:237] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d71fc57b804e8b9b7d1f6e448002f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42143 } } peers { permanent_uuid: "0494b540724f4ae88f279161bbbd5c05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } } peers { permanent_uuid: "721fcfa79c5847d1adae9d3dad5db772" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } }
I20260926 20:56:14.312623 5015 catalog_manager.cc:5495] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721fcfa79c5847d1adae9d3dad5db772 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "721fcfa79c5847d1adae9d3dad5db772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d71fc57b804e8b9b7d1f6e448002f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0494b540724f4ae88f279161bbbd5c05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "721fcfa79c5847d1adae9d3dad5db772" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42033 } health_report { overall_health: HEALTHY } } }
W20260926 20:56:14.322989 5204 tablet.cc:2369] T d437b34c420f4f13a8c43117797d04b1 P 0494b540724f4ae88f279161bbbd5c05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:56:14.365762 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.364317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.0.98.2:42033)
0926 20:56:14.364378 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:14.364381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:14.364452 (+ 71us) server_negotiation.cc:407] Connection header received
0926 20:56:14.364545 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:14.364547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:14.364567 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:14.364660 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:14.365421 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:14.365422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:14.365422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:14.365427 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:14.365446 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:14.365477 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:14.365541 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:14.365618 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 20:56:14.365633 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 20:56:14.367827 5157 raft_consensus.cc:1261] T d437b34c420f4f13a8c43117797d04b1 P 0494b540724f4ae88f279161bbbd5c05 [term 1 FOLLOWER]: Refusing update from remote peer 721fcfa79c5847d1adae9d3dad5db772: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:14.367827 5419 raft_consensus.cc:1261] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1 [term 1 FOLLOWER]: Refusing update from remote peer 721fcfa79c5847d1adae9d3dad5db772: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:14.368265 5489 consensus_queue.cc:1035] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73d71fc57b804e8b9b7d1f6e448002f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:14.368440 5486 consensus_queue.cc:1035] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0494b540724f4ae88f279161bbbd5c05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:14.371114 5492 mvcc.cc:205] Tried to move back new op lower bound from 7333708490207383552 to 7333708489981313024. Current Snapshot: MvccSnapshot[applied={T|T < 7333708490207383552}]
I20260926 20:56:14.371200 5493 mvcc.cc:205] Tried to move back new op lower bound from 7333708490207383552 to 7333708489981313024. Current Snapshot: MvccSnapshot[applied={T|T < 7333708490207383552}]
I20260926 20:56:14.371560 5494 mvcc.cc:205] Tried to move back new op lower bound from 7333708490207383552 to 7333708489981313024. Current Snapshot: MvccSnapshot[applied={T|T < 7333708490207383552}]
I20260926 20:56:14.373894 5014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:14.374017 5014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:14.374876 5014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ae5e41e94364a8d8d0376aba1416237: Sending DeleteTablet for 3 replicas of tablet d437b34c420f4f13a8c43117797d04b1
I20260926 20:56:14.375247 5268 tablet_service.cc:1514] Processing DeleteTablet for tablet d437b34c420f4f13a8c43117797d04b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:14 UTC) from {username='slave'} at 127.0.0.1:44442
I20260926 20:56:14.375270 5399 tablet_service.cc:1514] Processing DeleteTablet for tablet d437b34c420f4f13a8c43117797d04b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:14 UTC) from {username='slave'} at 127.0.0.1:57842
I20260926 20:56:14.375524 5137 tablet_service.cc:1514] Processing DeleteTablet for tablet d437b34c420f4f13a8c43117797d04b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:14 UTC) from {username='slave'} at 127.0.0.1:33818
I20260926 20:56:14.375752 5504 tablet_replica.cc:330] T d437b34c420f4f13a8c43117797d04b1 P 0494b540724f4ae88f279161bbbd5c05: stopping tablet replica
I20260926 20:56:14.375869 5504 raft_consensus.cc:2229] T d437b34c420f4f13a8c43117797d04b1 P 0494b540724f4ae88f279161bbbd5c05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:14.376012 5504 raft_consensus.cc:2258] T d437b34c420f4f13a8c43117797d04b1 P 0494b540724f4ae88f279161bbbd5c05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:14.376061 5503 tablet_replica.cc:330] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1: stopping tablet replica
I20260926 20:56:14.376138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 5075
I20260926 20:56:14.376164 5503 raft_consensus.cc:2229] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:14.376291 5503 raft_consensus.cc:2258] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:14.376462 5502 tablet_replica.cc:330] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772: stopping tablet replica
I20260926 20:56:14.376549 5502 raft_consensus.cc:2229] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:56:14.376667 5502 raft_consensus.cc:2258] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:14.377291 5502 ts_tablet_manager.cc:1905] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:14.377291 5503 ts_tablet_manager.cc:1905] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:14.379220 5503 ts_tablet_manager.cc:1918] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:14.379220 5502 ts_tablet_manager.cc:1918] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:14.379297 5502 log.cc:1196] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/d437b34c420f4f13a8c43117797d04b1
I20260926 20:56:14.379338 5503 log.cc:1196] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/d437b34c420f4f13a8c43117797d04b1
I20260926 20:56:14.379667 5502 ts_tablet_manager.cc:1939] T d437b34c420f4f13a8c43117797d04b1 P 721fcfa79c5847d1adae9d3dad5db772: Deleting consensus metadata
I20260926 20:56:14.379681 5503 ts_tablet_manager.cc:1939] T d437b34c420f4f13a8c43117797d04b1 P 73d71fc57b804e8b9b7d1f6e448002f1: Deleting consensus metadata
I20260926 20:56:14.380179 5002 catalog_manager.cc:4841] TS 73d71fc57b804e8b9b7d1f6e448002f1 (127.0.98.3:42143): tablet d437b34c420f4f13a8c43117797d04b1 (table test-table [id=a029ea04676a4317a0361c8139c7f034]) successfully deleted
I20260926 20:56:14.380300 5001 catalog_manager.cc:4841] TS 721fcfa79c5847d1adae9d3dad5db772 (127.0.98.2:42033): tablet d437b34c420f4f13a8c43117797d04b1 (table test-table [id=a029ea04676a4317a0361c8139c7f034]) successfully deleted
W20260926 20:56:14.384500 5000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46381 (error 108)
I20260926 20:56:14.385080 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.384930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46381 (local address 127.0.0.1:33824)
0926 20:56:14.384987 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:14.385043 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 20:56:14.385085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 5206
W20260926 20:56:14.385789 5000 catalog_manager.cc:4584] TS 0494b540724f4ae88f279161bbbd5c05 (127.0.98.1:46381): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d437b34c420f4f13a8c43117797d04b1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46381: connect: Connection refused (error 111)
I20260926 20:56:14.390141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 5337
I20260926 20:56:14.394943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 4984
2026-09-26T20:56:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (857 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 20:56:14.414599 392 mini_oidc.cc:152] Starting JWKS server
I20260926 20:56:14.415738 392 webserver.cc:460] Webserver started at https://127.0.0.1:36909/ using document root <none> and password file <none>
I20260926 20:56:14.415827 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:56:14.416046 392 webserver.cc:460] Webserver started at http://127.0.0.1:40529/ using document root <none> and password file <none>
2026-09-26T20:56:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:14Z Disabled control of system clock
I20260926 20:56:14.424602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44211
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44211
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36909/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/testchainca.cert with env {}
W20260926 20:56:14.510198 5511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:14.510531 5511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:14.510586 5511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:14.512017 5511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:14.512109 5511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:14.512132 5511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:14.512156 5511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:14.512197 5511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:56:14.512233 5511 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:56:14.512261 5511 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36909/jwks.json
I20260926 20:56:14.514173 5511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36909/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:14.514561 5511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:14.514830 5511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:14.520371 5516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:14.520455 5517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:14.520503 5511 server_base.cc:1034] running on GCE node
W20260926 20:56:14.520395 5519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:14.520849 5511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:14.521142 5511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:14.522323 5511 hybrid_clock.cc:648] HybridClock initialized: now 1790456174522298 us; error 48 us; skew 500 ppm
I20260926 20:56:14.523789 5511 webserver.cc:460] Webserver started at http://127.0.98.62:40353/ using document root <none> and password file <none>
I20260926 20:56:14.524085 5511 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:14.524163 5511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:14.524314 5511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:14.525436 5511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "a443a64be83f49e0b6ecac8488a32355"
format_stamp: "Formatted at 2026-09-26 20:56:14 on dist-test-slave-fl7g"
server_key: "ad108df98b878fd1345678b209469f80"
server_key_iv: "45e082f3da23d13b3bc749af8b602eac"
server_key_version: "encryptionkey@0"
I20260926 20:56:14.525852 5511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "a443a64be83f49e0b6ecac8488a32355"
format_stamp: "Formatted at 2026-09-26 20:56:14 on dist-test-slave-fl7g"
server_key: "ad108df98b878fd1345678b209469f80"
server_key_iv: "45e082f3da23d13b3bc749af8b602eac"
server_key_version: "encryptionkey@0"
I20260926 20:56:14.527155 5511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:56:14.527983 5525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:14.528177 5511 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:14.528278 5511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "a443a64be83f49e0b6ecac8488a32355"
format_stamp: "Formatted at 2026-09-26 20:56:14 on dist-test-slave-fl7g"
server_key: "ad108df98b878fd1345678b209469f80"
server_key_iv: "45e082f3da23d13b3bc749af8b602eac"
server_key_version: "encryptionkey@0"
I20260926 20:56:14.528368 5511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:14.536695 5511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:14.537046 5511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:14.537204 5511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:14.541388 5511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44211
I20260926 20:56:14.541435 5577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44211 every 8 connection(s)
I20260926 20:56:14.541803 5511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:14.542825 5578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:14.545328 5578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355: Bootstrap starting.
I20260926 20:56:14.545898 5578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:14.546146 5578 log.cc:824] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:14.546828 5578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355: No bootstrap required, opened a new log
I20260926 20:56:14.548360 5578 raft_consensus.cc:348] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a443a64be83f49e0b6ecac8488a32355" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44211 } }
I20260926 20:56:14.548597 5578 raft_consensus.cc:374] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:14.548627 5578 raft_consensus.cc:729] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a443a64be83f49e0b6ecac8488a32355, State: Initialized, Role: FOLLOWER
I20260926 20:56:14.548743 5578 consensus_queue.cc:260] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a443a64be83f49e0b6ecac8488a32355" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44211 } }
I20260926 20:56:14.548828 5578 raft_consensus.cc:388] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:14.548877 5578 raft_consensus.cc:482] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:14.548929 5578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:14.549221 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 5511
I20260926 20:56:14.549330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:56:14.549566 392 external_mini_cluster.cc:1428] Setting key 873aa7d3a1ada5fb1e7c5298236cb5aa
I20260926 20:56:14.549631 5578 raft_consensus.cc:504] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a443a64be83f49e0b6ecac8488a32355" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44211 } }
I20260926 20:56:14.549767 5578 leader_election.cc:302] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a443a64be83f49e0b6ecac8488a32355; no voters:
I20260926 20:56:14.549957 5578 leader_election.cc:288] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:14.550024 5581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:14.550164 5581 raft_consensus.cc:686] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355 [term 1 LEADER]: Becoming Leader. State: Replica: a443a64be83f49e0b6ecac8488a32355, State: Running, Role: LEADER
I20260926 20:56:14.550232 5578 sys_catalog.cc:564] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:14.550253 5581 consensus_queue.cc:237] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a443a64be83f49e0b6ecac8488a32355" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44211 } }
I20260926 20:56:14.550693 5582 sys_catalog.cc:455] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a443a64be83f49e0b6ecac8488a32355. Latest consensus state: current_term: 1 leader_uuid: "a443a64be83f49e0b6ecac8488a32355" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a443a64be83f49e0b6ecac8488a32355" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44211 } } }
I20260926 20:56:14.550776 5582 sys_catalog.cc:458] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:14.550916 5584 sys_catalog.cc:455] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a443a64be83f49e0b6ecac8488a32355" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a443a64be83f49e0b6ecac8488a32355" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44211 } } }
I20260926 20:56:14.550979 5584 sys_catalog.cc:458] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:14.551333 5587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:14.551884 5587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:14.553553 5587 catalog_manager.cc:1347] Generated new cluster ID: 8bcf468ecee94b248699edb4ba45b5d6
I20260926 20:56:14.553608 5587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:56:14.556090 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.550101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.0.98.62:44211)
0926 20:56:14.550499 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:14.550509 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:14.550543 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:56:14.550679 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:14.550687 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:14.550864 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:14.551392 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:14.551488 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.553140 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.554125 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.554575 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.554725 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:14.555434 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:14.555438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:14.555438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:14.555624 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:14.555656 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:14.555667 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:14.555747 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:14.555822 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 20:56:14.555901 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":59,"threads_started":1}
I20260926 20:56:14.557328 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:56:14.569162 5587 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:14.569905 5587 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:14.578347 5587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a443a64be83f49e0b6ecac8488a32355: Generated new TSK 0
I20260926 20:56:14.578574 5587 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 20:56:14.606822 5608 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 20:56:14.606945 5608 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 20:56:14.607223 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.590739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33298 (local address 127.0.98.62:44211)
0926 20:56:14.590789 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:14.590791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:14.590941 (+ 150us) server_negotiation.cc:407] Connection header received
0926 20:56:14.591053 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:14.591054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:14.591066 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:14.591155 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:56:14.591311 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.592088 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.593037 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.593353 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.593429 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:14.593695 (+ 266us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:56:14.606966 (+ 13271us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 20:56:14.607143 (+ 177us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33298: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":28}
W20260926 20:56:14.607292 5585 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33298: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 20:56:14.607657 5607 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:14.590657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44211 (local address 127.0.0.1:33298)
0926 20:56:14.590863 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:14.590873 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:14.590946 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:14.591173 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:14.591173 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:14.591175 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:56:14.591240 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:14.591242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.592200 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.592201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:14.592935 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:14.592938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.593455 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.593459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:14.593529 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:14.593556 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:56:14.607176 (+ 13620us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 20:56:14.607308 (+ 132us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44211: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":109,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:44211: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 20:56:14.608314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 5511
2026-09-26T20:56:14Z chronyd exiting
I20260926 20:56:14.644711 392 test_util.cc:175] -----------------------------------------------
I20260926 20:56:14.644804 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790456150796563-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (231 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 20:56:14.645605 392 mini_oidc.cc:152] Starting JWKS server
I20260926 20:56:14.646318 392 webserver.cc:460] Webserver started at https://127.0.0.1:41597/ using document root <none> and password file <none>
I20260926 20:56:14.646360 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:56:14.646543 392 webserver.cc:460] Webserver started at http://127.0.0.1:42775/ using document root <none> and password file <none>
2026-09-26T20:56:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:14Z Disabled control of system clock
I20260926 20:56:14.654366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45075
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45075
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41597/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/testchainca.cert with env {}
W20260926 20:56:14.735782 5614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:14.736012 5614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:14.736037 5614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:14.737313 5614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:14.737365 5614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:14.737377 5614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:14.737390 5614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:14.737402 5614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:56:14.737414 5614 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:56:14.737426 5614 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41597/jwks.json
I20260926 20:56:14.739105 5614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45075
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41597/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:14.739375 5614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:14.739602 5614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:14.744483 5622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:14.744513 5619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:14.744598 5614 server_base.cc:1034] running on GCE node
W20260926 20:56:14.744709 5620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:14.745003 5614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:14.745322 5614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:14.746507 5614 hybrid_clock.cc:648] HybridClock initialized: now 1790456174746481 us; error 33 us; skew 500 ppm
I20260926 20:56:14.747912 5614 webserver.cc:460] Webserver started at http://127.0.98.62:37263/ using document root <none> and password file <none>
I20260926 20:56:14.748191 5614 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:14.748261 5614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:14.748394 5614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:14.749543 5614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "d21fb5d16b6141298730ba9ae3a23302"
format_stamp: "Formatted at 2026-09-26 20:56:14 on dist-test-slave-fl7g"
server_key: "cf46a0336134e15fbe3fda9600f18695"
server_key_iv: "cb66d4b0fba491fb9bb22dd0f698b5fd"
server_key_version: "encryptionkey@0"
I20260926 20:56:14.749909 5614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "d21fb5d16b6141298730ba9ae3a23302"
format_stamp: "Formatted at 2026-09-26 20:56:14 on dist-test-slave-fl7g"
server_key: "cf46a0336134e15fbe3fda9600f18695"
server_key_iv: "cb66d4b0fba491fb9bb22dd0f698b5fd"
server_key_version: "encryptionkey@0"
I20260926 20:56:14.751242 5614 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:56:14.751997 5628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:14.752295 5614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:14.752377 5614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "d21fb5d16b6141298730ba9ae3a23302"
format_stamp: "Formatted at 2026-09-26 20:56:14 on dist-test-slave-fl7g"
server_key: "cf46a0336134e15fbe3fda9600f18695"
server_key_iv: "cb66d4b0fba491fb9bb22dd0f698b5fd"
server_key_version: "encryptionkey@0"
I20260926 20:56:14.752482 5614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:14.771638 5614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:14.771919 5614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:14.772030 5614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:14.775903 5614 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45075
I20260926 20:56:14.775959 5680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45075 every 8 connection(s)
I20260926 20:56:14.776327 5614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:14.777436 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:14.779078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 5614
I20260926 20:56:14.779161 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:56:14.779361 392 external_mini_cluster.cc:1428] Setting key e56c8a194b1ecb759415f0bc2adbacbf
I20260926 20:56:14.780521 5681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302: Bootstrap starting.
I20260926 20:56:14.781308 5681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:14.781651 5681 log.cc:824] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:14.782462 5681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302: No bootstrap required, opened a new log
I20260926 20:56:14.784922 5681 raft_consensus.cc:348] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21fb5d16b6141298730ba9ae3a23302" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45075 } }
I20260926 20:56:14.785105 5681 raft_consensus.cc:374] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:14.785157 5681 raft_consensus.cc:729] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d21fb5d16b6141298730ba9ae3a23302, State: Initialized, Role: FOLLOWER
I20260926 20:56:14.785305 5681 consensus_queue.cc:260] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21fb5d16b6141298730ba9ae3a23302" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45075 } }
I20260926 20:56:14.785398 5681 raft_consensus.cc:388] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:14.785434 5681 raft_consensus.cc:482] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:14.785485 5681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:14.786142 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.779939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.0.98.62:45075)
0926 20:56:14.780196 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:14.780206 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:14.780242 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:56:14.780845 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:14.780857 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:14.780962 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:14.781135 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:14.781305 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.782832 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.783705 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.784503 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.784662 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:14.785367 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:14.785371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:14.785371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:14.785616 (+ 245us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:14.785664 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:14.785681 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:14.785756 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:14.785831 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 20:56:14.785917 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":98,"threads_started":1}
I20260926 20:56:14.786326 5681 raft_consensus.cc:504] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21fb5d16b6141298730ba9ae3a23302" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45075 } }
I20260926 20:56:14.786450 5681 leader_election.cc:302] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d21fb5d16b6141298730ba9ae3a23302; no voters:
I20260926 20:56:14.786645 5681 leader_election.cc:288] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:14.786710 5686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:14.786919 5681 sys_catalog.cc:564] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:14.786962 5686 raft_consensus.cc:686] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302 [term 1 LEADER]: Becoming Leader. State: Replica: d21fb5d16b6141298730ba9ae3a23302, State: Running, Role: LEADER
I20260926 20:56:14.787055 5686 consensus_queue.cc:237] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21fb5d16b6141298730ba9ae3a23302" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45075 } }
I20260926 20:56:14.787209 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:56:14.788516 5688 sys_catalog.cc:455] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d21fb5d16b6141298730ba9ae3a23302. Latest consensus state: current_term: 1 leader_uuid: "d21fb5d16b6141298730ba9ae3a23302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21fb5d16b6141298730ba9ae3a23302" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45075 } } }
I20260926 20:56:14.788621 5688 sys_catalog.cc:458] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:14.788601 5687 sys_catalog.cc:455] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d21fb5d16b6141298730ba9ae3a23302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21fb5d16b6141298730ba9ae3a23302" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45075 } } }
I20260926 20:56:14.788691 5687 sys_catalog.cc:458] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302 [sys.catalog]: This master's current role is: LEADER
W20260926 20:56:14.789280 5700 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:56:14.789323 5700 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:56:14.789445 5701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:14.789674 5701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:14.791057 5701 catalog_manager.cc:1347] Generated new cluster ID: 4182133aadc943788ade7aed211ee5bb
I20260926 20:56:14.791121 5701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:56:14.817776 5701 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:14.818374 5701 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:14.821067 5701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d21fb5d16b6141298730ba9ae3a23302: Generated new TSK 0
I20260926 20:56:14.821264 5701 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 20:56:14.855443 5711 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 20:56:14.855547 5711 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 20:56:14.856094 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.832734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42752 (local address 127.0.98.62:45075)
0926 20:56:14.832818 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:14.832821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:14.832952 (+ 131us) server_negotiation.cc:407] Connection header received
0926 20:56:14.833026 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:14.833027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:14.833039 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:14.833121 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:56:14.833334 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.833879 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.835143 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.835356 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.835416 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:14.835632 (+ 216us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:56:14.855836 (+ 20204us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41597/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 20:56:14.855990 (+ 154us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42752: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41597/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 20:56:14.856215 5684 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42752: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41597/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 20:56:14.856300 5710 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:14.832669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45075 (local address 127.0.0.1:42752)
0926 20:56:14.832862 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:14.832874 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:14.832955 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:14.833139 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:14.833140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:14.833143 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:56:14.833208 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:14.833210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.833984 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.833985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:14.835027 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:14.835031 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.835434 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.835438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:14.835497 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:14.835503 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:56:14.856026 (+ 20523us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41597/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 20:56:14.856168 (+ 142us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45075: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41597/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":134,"threads_started":1}
I20260926 20:56:14.856681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 5614
2026-09-26T20:56:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (249 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 20:56:14.894994 392 mini_oidc.cc:152] Starting JWKS server
I20260926 20:56:14.895726 392 webserver.cc:460] Webserver started at https://127.0.0.1:41061/ using document root <none> and password file <none>
I20260926 20:56:14.895769 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:56:14.895926 392 webserver.cc:460] Webserver started at http://127.0.0.1:40833/ using document root <none> and password file <none>
2026-09-26T20:56:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:14Z Disabled control of system clock
I20260926 20:56:14.903967 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43119
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43119
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41061/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 20:56:14.998664 5717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:14.998888 5717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:14.998909 5717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:15.000062 5717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:15.000103 5717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:15.000116 5717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:15.000129 5717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:15.000142 5717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:56:15.000154 5717 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:56:15.000166 5717 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41061/jwks.json
I20260926 20:56:15.001623 5717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43119
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41061/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:15.001849 5717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:15.002112 5717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:15.007305 5725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:15.007285 5722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:15.007376 5723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:15.007882 5717 server_base.cc:1034] running on GCE node
I20260926 20:56:15.008198 5717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:15.008610 5717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:15.009796 5717 hybrid_clock.cc:648] HybridClock initialized: now 1790456175009775 us; error 33 us; skew 500 ppm
I20260926 20:56:15.011189 5717 webserver.cc:460] Webserver started at http://127.0.98.62:39521/ using document root <none> and password file <none>
I20260926 20:56:15.011482 5717 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:15.011533 5717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:15.011716 5717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:15.012912 5717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "096a1bd7392443b5b1431377e0af5f67"
format_stamp: "Formatted at 2026-09-26 20:56:15 on dist-test-slave-fl7g"
server_key: "c7967b425fe5ac6bbfacae9242f431e8"
server_key_iv: "61dbfdf00864cd4f90f8ce2207cb9d8e"
server_key_version: "encryptionkey@0"
I20260926 20:56:15.013291 5717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "096a1bd7392443b5b1431377e0af5f67"
format_stamp: "Formatted at 2026-09-26 20:56:15 on dist-test-slave-fl7g"
server_key: "c7967b425fe5ac6bbfacae9242f431e8"
server_key_iv: "61dbfdf00864cd4f90f8ce2207cb9d8e"
server_key_version: "encryptionkey@0"
I20260926 20:56:15.014777 5717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:56:15.015628 5731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:15.015868 5717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:15.015981 5717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "096a1bd7392443b5b1431377e0af5f67"
format_stamp: "Formatted at 2026-09-26 20:56:15 on dist-test-slave-fl7g"
server_key: "c7967b425fe5ac6bbfacae9242f431e8"
server_key_iv: "61dbfdf00864cd4f90f8ce2207cb9d8e"
server_key_version: "encryptionkey@0"
I20260926 20:56:15.016062 5717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:15.025852 5717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:15.026235 5717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:15.026410 5717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:15.030906 5717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43119
I20260926 20:56:15.030941 5783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43119 every 8 connection(s)
I20260926 20:56:15.031399 5717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:15.032177 5784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:15.035040 5784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67: Bootstrap starting.
I20260926 20:56:15.035712 5784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:15.035966 5784 log.cc:824] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:15.036746 5784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67: No bootstrap required, opened a new log
I20260926 20:56:15.038331 5784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096a1bd7392443b5b1431377e0af5f67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43119 } }
I20260926 20:56:15.038483 5784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:15.038506 5784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 096a1bd7392443b5b1431377e0af5f67, State: Initialized, Role: FOLLOWER
I20260926 20:56:15.038630 5784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096a1bd7392443b5b1431377e0af5f67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43119 } }
I20260926 20:56:15.038686 5784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:15.038707 5784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:15.038734 5784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:15.039058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 5717
I20260926 20:56:15.039155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:56:15.039327 392 external_mini_cluster.cc:1428] Setting key edbc516875cf8641958684b868de1bc2
I20260926 20:56:15.039382 5784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096a1bd7392443b5b1431377e0af5f67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43119 } }
I20260926 20:56:15.039525 5784 leader_election.cc:302] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 096a1bd7392443b5b1431377e0af5f67; no voters:
I20260926 20:56:15.039734 5784 leader_election.cc:288] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:15.039793 5787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:15.039976 5787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67 [term 1 LEADER]: Becoming Leader. State: Replica: 096a1bd7392443b5b1431377e0af5f67, State: Running, Role: LEADER
I20260926 20:56:15.040015 5784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:15.040081 5787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096a1bd7392443b5b1431377e0af5f67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43119 } }
I20260926 20:56:15.040465 5787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 096a1bd7392443b5b1431377e0af5f67. Latest consensus state: current_term: 1 leader_uuid: "096a1bd7392443b5b1431377e0af5f67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096a1bd7392443b5b1431377e0af5f67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43119 } } }
I20260926 20:56:15.040565 5787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:15.040629 5791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "096a1bd7392443b5b1431377e0af5f67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096a1bd7392443b5b1431377e0af5f67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43119 } } }
I20260926 20:56:15.040704 5791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:15.040783 5796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:15.041323 5796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:15.043090 5796 catalog_manager.cc:1347] Generated new cluster ID: 88ba3e6409394594b174bde4dc0b90bd
I20260926 20:56:15.043146 5796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:56:15.046415 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:15.039843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40512 (local address 127.0.98.62:43119)
0926 20:56:15.040263 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:15.040273 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:15.040308 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:56:15.040483 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:15.040491 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:15.040692 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:15.041287 (+ 595us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:15.041477 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:15.042945 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:15.044103 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:15.044756 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:15.044885 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:15.045676 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:15.045680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:15.045680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:15.045898 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:15.045943 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:15.045958 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:15.046031 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:15.046099 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 20:56:15.046185 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":76,"threads_started":1}
I20260926 20:56:15.047412 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:56:15.052657 5796 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:15.053341 5796 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:15.067212 5796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 096a1bd7392443b5b1431377e0af5f67: Generated new TSK 0
I20260926 20:56:15.067487 5796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:56:15.083588 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:15.079467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40520 (local address 127.0.98.62:43119)
0926 20:56:15.079532 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:15.079534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:15.079642 (+ 108us) server_negotiation.cc:407] Connection header received
0926 20:56:15.079706 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:15.079708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:15.079722 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:15.079809 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:56:15.079957 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:15.080804 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:15.081682 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:15.081995 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:15.082081 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:15.082299 (+ 218us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:56:15.083358 (+ 1059us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41061/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 20:56:15.083501 (+ 143us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40520: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41061/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 20:56:15.083671 5790 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40520: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41061/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 20:56:15.085775 5813 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:15.079361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43119 (local address 127.0.0.1:40520)
0926 20:56:15.079558 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:15.079572 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:15.079642 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:15.079814 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:15.079814 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:15.079816 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:56:15.079889 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:15.079891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:15.080936 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:15.080937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:15.081586 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:15.081589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:15.082096 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:15.082105 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:15.082169 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:15.082175 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:56:15.083479 (+ 1304us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41061/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 20:56:15.085668 (+ 2189us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43119: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41061/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":1917,"thread_start_us":125,"threads_started":1}
I20260926 20:56:15.086265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 5717
E20260926 20:56:15.096447 5814 webserver.cc:533] Webserver: sslize failed:
E20260926 20:56:15.096542 5814 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T20:56:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (229 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T20:56:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:15Z Disabled control of system clock
I20260926 20:56:15.132829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43673
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43673
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/testchainca.cert with env {}
W20260926 20:56:15.215613 5818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:15.215855 5818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:15.215880 5818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:15.217303 5818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:15.217356 5818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:15.217370 5818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:15.217383 5818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:15.217396 5818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:56:15.217409 5818 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:56:15.217464 5818 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 20:56:15.219025 5818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43673
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:15.219310 5818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:15.219555 5818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:15.224880 5826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:15.224880 5823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:15.225005 5824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:15.225095 5818 server_base.cc:1034] running on GCE node
I20260926 20:56:15.225450 5818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:15.225818 5818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:15.227001 5818 hybrid_clock.cc:648] HybridClock initialized: now 1790456175226981 us; error 42 us; skew 500 ppm
I20260926 20:56:15.228297 5818 webserver.cc:460] Webserver started at http://127.0.98.62:35075/ using document root <none> and password file <none>
I20260926 20:56:15.228590 5818 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:15.228667 5818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:15.228811 5818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:15.229928 5818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "9d946515c8ce4b3aaf74dc4e72005c15"
format_stamp: "Formatted at 2026-09-26 20:56:15 on dist-test-slave-fl7g"
server_key: "f89db0a5db6a4c23ab0033bdb236fc07"
server_key_iv: "3301809a1bba6268a672ce34c5c83863"
server_key_version: "encryptionkey@0"
I20260926 20:56:15.230320 5818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "9d946515c8ce4b3aaf74dc4e72005c15"
format_stamp: "Formatted at 2026-09-26 20:56:15 on dist-test-slave-fl7g"
server_key: "f89db0a5db6a4c23ab0033bdb236fc07"
server_key_iv: "3301809a1bba6268a672ce34c5c83863"
server_key_version: "encryptionkey@0"
I20260926 20:56:15.231848 5818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:56:15.232648 5832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:15.232829 5818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:15.232939 5818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "9d946515c8ce4b3aaf74dc4e72005c15"
format_stamp: "Formatted at 2026-09-26 20:56:15 on dist-test-slave-fl7g"
server_key: "f89db0a5db6a4c23ab0033bdb236fc07"
server_key_iv: "3301809a1bba6268a672ce34c5c83863"
server_key_version: "encryptionkey@0"
I20260926 20:56:15.233018 5818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:15.251219 5818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:15.251665 5818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:15.251868 5818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:15.256027 5818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43673
I20260926 20:56:15.256122 5884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43673 every 8 connection(s)
I20260926 20:56:15.256570 5818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:15.257462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 5818
I20260926 20:56:15.257611 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:56:15.257728 5885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:15.257838 392 external_mini_cluster.cc:1428] Setting key d2b79a8ff1406609812a1997981cd62d
I20260926 20:56:15.261276 5885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15: Bootstrap starting.
I20260926 20:56:15.261927 5885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:15.262308 5885 log.cc:824] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:15.263067 5885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15: No bootstrap required, opened a new log
I20260926 20:56:15.263460 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:15.258386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52312 (local address 127.0.98.62:43673)
0926 20:56:15.258595 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:15.258602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:15.258628 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:56:15.259013 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:15.259020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:15.259086 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:15.259204 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:15.259432 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:15.260330 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:15.261475 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:15.261913 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:15.262047 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:15.262820 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:15.262825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:15.262826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:15.263000 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:15.263053 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:15.263064 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:15.263144 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:15.263212 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 20:56:15.263292 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260926 20:56:15.264473 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:56:15.265559 5885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d946515c8ce4b3aaf74dc4e72005c15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43673 } }
I20260926 20:56:15.265727 5885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:15.265774 5885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d946515c8ce4b3aaf74dc4e72005c15, State: Initialized, Role: FOLLOWER
I20260926 20:56:15.265893 5885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d946515c8ce4b3aaf74dc4e72005c15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43673 } }
I20260926 20:56:15.265995 5885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:15.266036 5885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:15.266086 5885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:15.266667 5885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d946515c8ce4b3aaf74dc4e72005c15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43673 } }
I20260926 20:56:15.266770 5885 leader_election.cc:302] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d946515c8ce4b3aaf74dc4e72005c15; no voters:
I20260926 20:56:15.266959 5885 leader_election.cc:288] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:15.267014 5890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:15.267176 5890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15 [term 1 LEADER]: Becoming Leader. State: Replica: 9d946515c8ce4b3aaf74dc4e72005c15, State: Running, Role: LEADER
I20260926 20:56:15.267239 5885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:15.267277 5890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d946515c8ce4b3aaf74dc4e72005c15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43673 } }
I20260926 20:56:15.267643 5891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d946515c8ce4b3aaf74dc4e72005c15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d946515c8ce4b3aaf74dc4e72005c15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43673 } } }
I20260926 20:56:15.267643 5892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d946515c8ce4b3aaf74dc4e72005c15. Latest consensus state: current_term: 1 leader_uuid: "9d946515c8ce4b3aaf74dc4e72005c15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d946515c8ce4b3aaf74dc4e72005c15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43673 } } }
I20260926 20:56:15.267761 5891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:15.267769 5892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:15.267980 5898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:15.268584 5898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:15.270324 5898 catalog_manager.cc:1347] Generated new cluster ID: 6f7ee48683da4c2692ab12a0910d0dc6
I20260926 20:56:15.270391 5898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:56:15.289652 5898 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:15.290302 5898 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:15.295145 5898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d946515c8ce4b3aaf74dc4e72005c15: Generated new TSK 0
I20260926 20:56:15.295408 5898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:56:15.377254 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:15.302716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52320 (local address 127.0.98.62:43673)
0926 20:56:15.302783 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:15.302785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:15.302907 (+ 122us) server_negotiation.cc:407] Connection header received
0926 20:56:15.302985 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:15.302986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:15.302999 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:15.303086 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:56:15.303280 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:15.304091 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:15.305011 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:15.305338 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:15.305415 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:15.305694 (+ 279us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:56:15.376965 (+ 71271us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 20:56:15.377173 (+ 208us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52320: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 20:56:15.377343 5888 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52320: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 20:56:15.377388 5914 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:15.302594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43673 (local address 127.0.0.1:52320)
0926 20:56:15.302834 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:15.302844 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:15.302907 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:15.303131 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:15.303132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:15.303134 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:56:15.303201 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:15.303203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:15.304187 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:15.304188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:15.304931 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:15.304934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:15.305503 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:15.305509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:15.305564 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:15.305570 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:56:15.377142 (+ 71572us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 20:56:15.377268 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43673: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":139,"threads_started":1}
I20260926 20:56:15.377942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 5818
2026-09-26T20:56:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (270 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (91 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (91 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:56:15 dist-test-slave-fl7g krb5kdc[5926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:56:15 dist-test-slave-fl7g krb5kdc[5926](info): set up 2 sockets
Sep 26 20:56:15 dist-test-slave-fl7g krb5kdc[5926](info): commencing operation
krb5kdc: starting...
W20260926 20:56:16.604722 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:56:16 dist-test-slave-fl7g krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:56:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:56:16.727941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46293
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:16.811040 5939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:16.811283 5939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:16.811309 5939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:16.812721 5939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:16.812775 5939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:16.812790 5939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:16.812803 5939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:16.812820 5939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:16.814487 5939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:16.814783 5939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:16.815028 5939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:16.820194 5944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:16.820201 5947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:16.820188 5945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:16.820379 5939 server_base.cc:1034] running on GCE node
I20260926 20:56:16.820791 5939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:16.821130 5939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:16.822293 5939 hybrid_clock.cc:648] HybridClock initialized: now 1790456176822274 us; error 41 us; skew 500 ppm
Sep 26 20:56:16 dist-test-slave-fl7g krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:16.824921 5939 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:56:16.825485 5939 webserver.cc:460] Webserver started at http://127.0.98.62:43743/ using document root <none> and password file <none>
I20260926 20:56:16.825698 5939 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:16.825742 5939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:16.825834 5939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:16.826894 5939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "954410d827c542b6870d0492f34bb807"
format_stamp: "Formatted at 2026-09-26 20:56:16 on dist-test-slave-fl7g"
server_key: "34a0fefd05ca2c5dd23210d36c3a1ce3"
server_key_iv: "70a4085bb17813782d07aef24da826f0"
server_key_version: "encryptionkey@0"
I20260926 20:56:16.827195 5939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "954410d827c542b6870d0492f34bb807"
format_stamp: "Formatted at 2026-09-26 20:56:16 on dist-test-slave-fl7g"
server_key: "34a0fefd05ca2c5dd23210d36c3a1ce3"
server_key_iv: "70a4085bb17813782d07aef24da826f0"
server_key_version: "encryptionkey@0"
I20260926 20:56:16.828581 5939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:56:16.829309 5954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:16.829483 5939 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:16.829607 5939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "954410d827c542b6870d0492f34bb807"
format_stamp: "Formatted at 2026-09-26 20:56:16 on dist-test-slave-fl7g"
server_key: "34a0fefd05ca2c5dd23210d36c3a1ce3"
server_key_iv: "70a4085bb17813782d07aef24da826f0"
server_key_version: "encryptionkey@0"
I20260926 20:56:16.829691 5939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:16.840062 5939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:16.842360 5939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:16.842543 5939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:16.846783 5939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46293
I20260926 20:56:16.846832 6006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46293 every 8 connection(s)
I20260926 20:56:16.847369 5939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:16.848057 6007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:16.850451 6007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807: Bootstrap starting.
I20260926 20:56:16.851104 6007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:16.851408 6007 log.cc:824] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:16.852177 6007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807: No bootstrap required, opened a new log
I20260926 20:56:16.853231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 5939
I20260926 20:56:16.853351 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:56:16.853556 392 external_mini_cluster.cc:1428] Setting key 1e8ad4d72fe00677f8183af9461036c9
I20260926 20:56:16.854488 6007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "954410d827c542b6870d0492f34bb807" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46293 } }
I20260926 20:56:16.854655 6007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:16.854691 6007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954410d827c542b6870d0492f34bb807, State: Initialized, Role: FOLLOWER
I20260926 20:56:16.854826 6007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "954410d827c542b6870d0492f34bb807" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46293 } }
I20260926 20:56:16.854925 6007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:16.854962 6007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:16.855024 6007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:16.855734 6007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "954410d827c542b6870d0492f34bb807" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46293 } }
I20260926 20:56:16.855825 6007 leader_election.cc:302] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 954410d827c542b6870d0492f34bb807; no voters:
I20260926 20:56:16.856019 6007 leader_election.cc:288] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:16.856074 6012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:16.856226 6012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807 [term 1 LEADER]: Becoming Leader. State: Replica: 954410d827c542b6870d0492f34bb807, State: Running, Role: LEADER
I20260926 20:56:16.856314 6012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "954410d827c542b6870d0492f34bb807" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46293 } }
I20260926 20:56:16.856343 6007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:16.856645 6012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 954410d827c542b6870d0492f34bb807. Latest consensus state: current_term: 1 leader_uuid: "954410d827c542b6870d0492f34bb807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "954410d827c542b6870d0492f34bb807" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46293 } } }
I20260926 20:56:16.856724 6012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:16.856676 6014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "954410d827c542b6870d0492f34bb807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "954410d827c542b6870d0492f34bb807" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46293 } } }
I20260926 20:56:16.856873 6014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:16.856925 6016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:16.857445 6016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:16.859586 6016 catalog_manager.cc:1347] Generated new cluster ID: f38a0d1a7b6346e09cde5239e6a5bb5e
I20260926 20:56:16.859645 6016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:56:16 dist-test-slave-fl7g krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:16.865446 6011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:16.854124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50272 (local address 127.0.98.62:46293)
0926 20:56:16.854321 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:16.854330 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:16.854358 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:56:16.854457 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:16.854463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:16.854537 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:16.854673 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:16.855426 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:16.856790 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:16.857973 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:16.858601 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:16.858730 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:16.861622 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:16.861626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:16.861627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:16.861639 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:16.863652 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:16.864111 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:16.864113 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:16.864114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:16.864231 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:16.864488 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:16.864489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:16.864489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:16.864792 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:16.864940 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:16.865083 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 20:56:16.865215 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
I20260926 20:56:16.868225 6016 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:16.868995 6016 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:16.874595 6016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 954410d827c542b6870d0492f34bb807: Generated new TSK 0
I20260926 20:56:16.874794 6016 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:56:16.904635 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46293
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:16.993178 6033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:16.993435 6033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:16.993477 6033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:16.994679 6033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:16.994755 6033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:16.994800 6033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:56:16.996379 6033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:16.996673 6033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:16.996909 6033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:17.002084 6038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:17.002117 6041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:17.002117 6039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:17.002375 6033 server_base.cc:1034] running on GCE node
I20260926 20:56:17.002610 6033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:17.002885 6033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:17.004082 6033 hybrid_clock.cc:648] HybridClock initialized: now 1790456177004077 us; error 39 us; skew 500 ppm
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:17.006832 6033 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:56:17.007285 6033 webserver.cc:460] Webserver started at http://127.0.98.1:45151/ using document root <none> and password file <none>
I20260926 20:56:17.007558 6033 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:17.007632 6033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:17.007781 6033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:17.008955 6033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "4674bb2ec8d04654a0c53452741851c7"
format_stamp: "Formatted at 2026-09-26 20:56:17 on dist-test-slave-fl7g"
server_key: "6c729751fbd81b59ea2cd401b372b2e0"
server_key_iv: "3d3aa776197648386efd182aae058918"
server_key_version: "encryptionkey@0"
I20260926 20:56:17.009332 6033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4674bb2ec8d04654a0c53452741851c7"
format_stamp: "Formatted at 2026-09-26 20:56:17 on dist-test-slave-fl7g"
server_key: "6c729751fbd81b59ea2cd401b372b2e0"
server_key_iv: "3d3aa776197648386efd182aae058918"
server_key_version: "encryptionkey@0"
I20260926 20:56:17.010847 6033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:56:17.011732 6048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:17.012082 6033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:17.012190 6033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "4674bb2ec8d04654a0c53452741851c7"
format_stamp: "Formatted at 2026-09-26 20:56:17 on dist-test-slave-fl7g"
server_key: "6c729751fbd81b59ea2cd401b372b2e0"
server_key_iv: "3d3aa776197648386efd182aae058918"
server_key_version: "encryptionkey@0"
I20260926 20:56:17.012279 6033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:17.040042 6033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:17.042519 6033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:17.042734 6033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:17.043020 6033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:17.043360 6033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:17.043418 6033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:17.043460 6033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:17.043504 6033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:17.049693 6033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44919
I20260926 20:56:17.049734 6161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44919 every 8 connection(s)
I20260926 20:56:17.050120 6033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:17.060322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 6033
I20260926 20:56:17.060492 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:56:17.060727 392 external_mini_cluster.cc:1428] Setting key 4658bd7bd1f23173c006fe2b995898ca
I20260926 20:56:17.060897 6011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:17.050887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39715 (local address 127.0.98.62:46293)
0926 20:56:17.050933 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:17.050935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:17.051182 (+ 247us) server_negotiation.cc:407] Connection header received
0926 20:56:17.052124 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:17.052126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:17.052144 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:17.052220 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:17.052972 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:17.053471 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:17.054565 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:17.054780 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:17.054847 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:17.057204 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:17.057206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:17.057206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:17.057215 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:17.059087 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:17.059597 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:17.059598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:17.059598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:17.059668 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:17.060006 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:17.060007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:17.060007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:17.060148 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:17.060255 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:17.060671 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 20:56:17.060789 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:56:17.061153 6164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:17.050829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46293 (local address 127.0.98.1:39715)
0926 20:56:17.051010 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:17.051054 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:17.052006 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:17.052301 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:17.052305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:17.052659 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:17.052862 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:17.052866 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:17.053604 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:17.053605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:17.054450 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:17.054452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:17.054866 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:17.054867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:17.054968 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:17.055496 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:17.055504 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:17.057084 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:17.059200 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:17.059208 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:17.059210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:17.059494 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:17.059764 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:17.059765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:17.059765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:17.059905 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:17.060258 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:17.060267 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:17.060494 (+ 227us) client_negotiation.cc:769] Sending connection context
0926 20:56:17.060650 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 20:56:17.060853 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260926 20:56:17.061820 6162 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46293
I20260926 20:56:17.061937 6162 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:17.062137 6162 heartbeater.cc:511] Master 127.0.98.62:46293 requested a full tablet report, sending...
I20260926 20:56:17.062747 5971 ts_manager.cc:194] Registered new tserver with Master: 4674bb2ec8d04654a0c53452741851c7 (127.0.98.1:44919)
I20260926 20:56:17.063504 5971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39715
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:56:17.088651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46293
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:17.171332 6167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:17.171550 6167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:17.171576 6167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:17.172719 6167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:17.172760 6167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:17.172786 6167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:56:17.174324 6167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:17.174525 6167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:17.174714 6167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:17.179847 6175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:17.179847 6172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:17.179865 6173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:17.180019 6167 server_base.cc:1034] running on GCE node
I20260926 20:56:17.180363 6167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:17.180684 6167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:17.181850 6167 hybrid_clock.cc:648] HybridClock initialized: now 1790456177181840 us; error 47 us; skew 500 ppm
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:17.184497 6167 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:56:17.185002 6167 webserver.cc:460] Webserver started at http://127.0.98.2:46349/ using document root <none> and password file <none>
I20260926 20:56:17.185256 6167 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:17.185323 6167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:17.185474 6167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:17.186561 6167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "477021f6418f40a6be67e517795948e6"
format_stamp: "Formatted at 2026-09-26 20:56:17 on dist-test-slave-fl7g"
server_key: "641d45689dc0097d174e8c5098722231"
server_key_iv: "a682059d07d331ded2540e08d34c28cd"
server_key_version: "encryptionkey@0"
I20260926 20:56:17.186935 6167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "477021f6418f40a6be67e517795948e6"
format_stamp: "Formatted at 2026-09-26 20:56:17 on dist-test-slave-fl7g"
server_key: "641d45689dc0097d174e8c5098722231"
server_key_iv: "a682059d07d331ded2540e08d34c28cd"
server_key_version: "encryptionkey@0"
I20260926 20:56:17.188318 6167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:56:17.189229 6182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:17.189426 6167 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:17.189529 6167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "477021f6418f40a6be67e517795948e6"
format_stamp: "Formatted at 2026-09-26 20:56:17 on dist-test-slave-fl7g"
server_key: "641d45689dc0097d174e8c5098722231"
server_key_iv: "a682059d07d331ded2540e08d34c28cd"
server_key_version: "encryptionkey@0"
I20260926 20:56:17.189620 6167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:17.204255 6167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:17.206481 6167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:17.206668 6167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:17.206939 6167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:17.207262 6167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:17.207317 6167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:17.207357 6167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:17.207403 6167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:17.213539 6167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41671
I20260926 20:56:17.213582 6295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41671 every 8 connection(s)
I20260926 20:56:17.213932 6167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:17.223155 6011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:17.214715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50691 (local address 127.0.98.62:46293)
0926 20:56:17.214782 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:17.214784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:17.215063 (+ 279us) server_negotiation.cc:407] Connection header received
0926 20:56:17.216064 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:17.216066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:17.216077 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:17.216136 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:17.216817 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:17.217325 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:17.218282 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:17.218467 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:17.218524 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:17.220535 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:17.220537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:17.220537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:17.220544 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:17.222029 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:17.222381 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:17.222382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:17.222382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:17.222435 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:17.222644 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:17.222644 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:17.222645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:17.222746 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:17.222796 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:17.223022 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 20:56:17.223086 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:56:17.223366 6298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:17.214711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46293 (local address 127.0.98.2:50691)
0926 20:56:17.214907 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:17.214960 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:17.215955 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:17.216205 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:17.216208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:17.216536 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:17.216734 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:17.216738 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:17.217407 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:17.217407 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:17.218204 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:17.218206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:17.218542 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:17.218543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:17.218641 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:17.219115 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:17.219123 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:17.220451 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:17.222102 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:17.222107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:17.222110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:17.222317 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:17.222491 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:17.222491 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:17.222492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:17.222588 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:17.222799 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:17.222803 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:17.222912 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 20:56:17.222993 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 20:56:17.223117 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":93,"threads_started":1}
I20260926 20:56:17.223745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 6167
I20260926 20:56:17.223876 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:56:17.224027 6296 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46293
I20260926 20:56:17.224124 6296 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:17.224138 392 external_mini_cluster.cc:1428] Setting key 4e376f42b7ea23573d64a67ab258081b
I20260926 20:56:17.224330 6296 heartbeater.cc:511] Master 127.0.98.62:46293 requested a full tablet report, sending...
I20260926 20:56:17.224809 5971 ts_manager.cc:194] Registered new tserver with Master: 477021f6418f40a6be67e517795948e6 (127.0.98.2:41671)
I20260926 20:56:17.225189 5971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50691
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:56:17.251387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46293
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:17.334421 6301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:17.334647 6301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:17.334671 6301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:17.335871 6301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:17.335917 6301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:17.335944 6301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:56:17.337643 6301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:17.337911 6301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:17.338164 6301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:17.343384 6309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:17.343374 6306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:17.343374 6307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:17.343570 6301 server_base.cc:1034] running on GCE node
I20260926 20:56:17.343904 6301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:17.344151 6301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:17.345295 6301 hybrid_clock.cc:648] HybridClock initialized: now 1790456177345279 us; error 35 us; skew 500 ppm
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:17.347827 6301 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:56:17.348276 6301 webserver.cc:460] Webserver started at http://127.0.98.3:36237/ using document root <none> and password file <none>
I20260926 20:56:17.348582 6301 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:17.348656 6301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:17.348798 6301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:17.349879 6301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "3482fc684a9c496d83ac6d84b6f8c9bd"
format_stamp: "Formatted at 2026-09-26 20:56:17 on dist-test-slave-fl7g"
server_key: "4634811e478b01c63aa642ec6029c7a1"
server_key_iv: "5c051a3e6e9bcccb519f00a8391da287"
server_key_version: "encryptionkey@0"
I20260926 20:56:17.350272 6301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3482fc684a9c496d83ac6d84b6f8c9bd"
format_stamp: "Formatted at 2026-09-26 20:56:17 on dist-test-slave-fl7g"
server_key: "4634811e478b01c63aa642ec6029c7a1"
server_key_iv: "5c051a3e6e9bcccb519f00a8391da287"
server_key_version: "encryptionkey@0"
I20260926 20:56:17.351657 6301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:56:17.352510 6316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:17.352748 6301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:17.352881 6301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "3482fc684a9c496d83ac6d84b6f8c9bd"
format_stamp: "Formatted at 2026-09-26 20:56:17 on dist-test-slave-fl7g"
server_key: "4634811e478b01c63aa642ec6029c7a1"
server_key_iv: "5c051a3e6e9bcccb519f00a8391da287"
server_key_version: "encryptionkey@0"
I20260926 20:56:17.352970 6301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:17.362805 6301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:17.365083 6301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:17.365255 6301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:17.365495 6301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:17.365830 6301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:17.365888 6301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:17.365933 6301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:17.365958 6301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:17.371905 6301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33505
I20260926 20:56:17.371945 6429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33505 every 8 connection(s)
I20260926 20:56:17.372285 6301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:56:17.376505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 6301
I20260926 20:56:17.376632 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:56:17.376821 392 external_mini_cluster.cc:1428] Setting key 6c1eab346da12bec108c68c64a03ed8b
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:17.382040 6011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:17.373123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36631 (local address 127.0.98.62:46293)
0926 20:56:17.373184 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:17.373186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:17.373376 (+ 190us) server_negotiation.cc:407] Connection header received
0926 20:56:17.374355 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:17.374356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:17.374371 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:17.374439 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:17.375178 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:17.375724 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:17.376890 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:17.377103 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:17.377188 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:17.379511 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:17.379512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:17.379513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:17.379520 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:17.380828 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:17.381230 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:17.381230 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:17.381231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:17.381278 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:17.381504 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:17.381504 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:17.381504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:17.381619 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:17.381670 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:17.381918 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 20:56:17.381976 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:56:17.382311 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:17.373091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46293 (local address 127.0.98.3:36631)
0926 20:56:17.373242 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:17.373285 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:17.374217 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:17.374534 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:17.374539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:17.374832 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:17.375090 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:17.375095 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:17.375809 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:17.375810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:17.376787 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:17.376790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:17.377243 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:17.377245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:17.377366 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:17.378014 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:17.378022 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:17.379427 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:17.380901 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:17.380907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:17.380909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:17.381162 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:17.381336 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:17.381336 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:17.381337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:17.381447 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:17.381676 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:17.381680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:17.381803 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 20:56:17.381893 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:56:17.382043 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260926 20:56:17.382804 6430 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46293
I20260926 20:56:17.382915 6430 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:17.383114 6430 heartbeater.cc:511] Master 127.0.98.62:46293 requested a full tablet report, sending...
I20260926 20:56:17.383582 5971 ts_manager.cc:194] Registered new tserver with Master: 3482fc684a9c496d83ac6d84b6f8c9bd (127.0.98.3:33505)
I20260926 20:56:17.383965 5971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36631
I20260926 20:56:17.388710 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/test-admin.keytab.
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:17.406167 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 20:56:17.407806 6437 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:17.407060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35748 (local address 127.0.98.1:44919)
0926 20:56:17.407242 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:17.407246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:17.407257 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:56:17.407282 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:17.407284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:17.407298 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:17.407368 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:17.407721 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35748: BlockingRecv error: recv got EOF from 127.0.0.1:35748 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":103,"threads_started":1}
W20260926 20:56:17.407980 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/krb5cc)
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:17.409080 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:17.409964 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:56:17.418561 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:17.410535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35756 (local address 127.0.98.1:44919)
0926 20:56:17.410603 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:17.410605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:17.410672 (+ 67us) server_negotiation.cc:407] Connection header received
0926 20:56:17.410716 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:17.410717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:17.410728 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:17.410782 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:17.411160 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:17.411973 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:17.412868 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:17.413232 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:17.413297 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:17.415580 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:17.415589 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:17.415589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:17.415596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:17.417467 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:17.417821 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:17.417823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:17.417824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:17.417881 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:17.418043 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:17.418043 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:17.418044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:17.418179 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:17.418294 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:17.418348 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 20:56:17.418410 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 20:56:17.420608 6437 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:17.419879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35764 (local address 127.0.98.1:44919)
0926 20:56:17.419944 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:17.419946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:17.420015 (+ 69us) server_negotiation.cc:407] Connection header received
0926 20:56:17.420095 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:17.420096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:17.420107 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:17.420158 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:17.420554 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35764: BlockingRecv error: recv got EOF from 127.0.0.1:35764 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 20:56:17.420672 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:17.421703 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:17.422596 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:56:17.430368 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:17.423134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35766 (local address 127.0.98.1:44919)
0926 20:56:17.423198 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:17.423200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:17.423261 (+ 61us) server_negotiation.cc:407] Connection header received
0926 20:56:17.423327 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:17.423328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:17.423340 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:17.423415 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:17.423813 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:17.424361 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:17.425116 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:17.425322 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:17.425412 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:17.427842 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:17.427844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:17.427844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:17.427851 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:17.429257 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:17.429632 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:17.429633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:17.429633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:17.429685 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:17.429859 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:17.429860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:17.429860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:17.429955 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:17.430024 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:17.430148 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 20:56:17.430215 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 20:56:17.432070 6437 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:17.431419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35780 (local address 127.0.98.1:44919)
0926 20:56:17.431464 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:17.431466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:17.431518 (+ 52us) server_negotiation.cc:407] Connection header received
0926 20:56:17.431576 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:17.431576 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:17.431585 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:17.431643 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:17.431999 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35780: BlockingRecv error: recv got EOF from 127.0.0.1:35780 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 20:56:17.432176 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790456150796563-392-0/krb5kdc/krb5cc)
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:17.433439 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:17.434434 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:56:17.442205 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:17.435073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35782 (local address 127.0.98.1:44919)
0926 20:56:17.435117 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:17.435119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:17.435170 (+ 51us) server_negotiation.cc:407] Connection header received
0926 20:56:17.435228 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:17.435229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:17.435238 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:17.435292 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:17.435733 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:17.436252 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:17.437110 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:17.437292 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:17.437352 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:17.439575 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:17.439576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:17.439577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:17.439584 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:17.441077 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:17.441402 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:17.441403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:17.441403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:17.441453 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:17.441622 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:17.441623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:17.441623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:17.441715 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:17.441786 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:17.441982 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 20:56:17.442080 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:56:17.443089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 6033
I20260926 20:56:17.448383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 6167
I20260926 20:56:17.452833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 6301
I20260926 20:56:17.457267 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 5939
2026-09-26T20:56:17Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1898 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[6450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[6450](info): set up 2 sockets
Sep 26 20:56:17 dist-test-slave-fl7g krb5kdc[6450](info): commencing operation
krb5kdc: starting...
W20260926 20:56:18.496816 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:56:18 dist-test-slave-fl7g krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:56:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 20:56:18.621762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44429
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:18.715610 6463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:18.715847 6463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:18.715871 6463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:18.717197 6463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:18.717255 6463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:18.717270 6463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:18.717283 6463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:18.717295 6463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:18.718892 6463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:18.719182 6463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:18.719432 6463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:18.724733 6468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:18.724747 6469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:18.724741 6471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:18.724912 6463 server_base.cc:1034] running on GCE node
I20260926 20:56:18.725236 6463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:18.725548 6463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:18.726760 6463 hybrid_clock.cc:648] HybridClock initialized: now 1790456178726758 us; error 49 us; skew 500 ppm
Sep 26 20:56:18 dist-test-slave-fl7g krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456178, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:18.729400 6463 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 20:56:18.729898 6463 webserver.cc:460] Webserver started at http://127.0.98.62:46547/ using document root <none> and password file <none>
I20260926 20:56:18.730306 6463 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:18.730386 6463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:18.730536 6463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:18.731735 6463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "896ea7692253410faae3e0185a54e566"
format_stamp: "Formatted at 2026-09-26 20:56:18 on dist-test-slave-fl7g"
server_key: "894ffa18684a010751053db5d9d2fd0d"
server_key_iv: "a363b714c9fb9afe4ae9e671659b64ff"
server_key_version: "encryptionkey@0"
I20260926 20:56:18.732115 6463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "896ea7692253410faae3e0185a54e566"
format_stamp: "Formatted at 2026-09-26 20:56:18 on dist-test-slave-fl7g"
server_key: "894ffa18684a010751053db5d9d2fd0d"
server_key_iv: "a363b714c9fb9afe4ae9e671659b64ff"
server_key_version: "encryptionkey@0"
I20260926 20:56:18.733637 6463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:56:18.734344 6478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:18.734524 6463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:18.734619 6463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "896ea7692253410faae3e0185a54e566"
format_stamp: "Formatted at 2026-09-26 20:56:18 on dist-test-slave-fl7g"
server_key: "894ffa18684a010751053db5d9d2fd0d"
server_key_iv: "a363b714c9fb9afe4ae9e671659b64ff"
server_key_version: "encryptionkey@0"
I20260926 20:56:18.734699 6463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:18.749771 6463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:18.752223 6463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:18.752388 6463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:18.756142 6463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44429
I20260926 20:56:18.756201 6530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44429 every 8 connection(s)
I20260926 20:56:18.756582 6463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:18.757063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 6463
I20260926 20:56:18.757189 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:56:18.757467 392 external_mini_cluster.cc:1428] Setting key a365d03242602b2d7b2f179ff3f8d727
I20260926 20:56:18.757918 6531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:18.761327 6531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566: Bootstrap starting.
I20260926 20:56:18.761974 6531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:18.762277 6531 log.cc:824] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:18.762933 6531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566: No bootstrap required, opened a new log
Sep 26 20:56:18 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:56:18.764964 6531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } }
I20260926 20:56:18.765132 6531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:18.765178 6531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 896ea7692253410faae3e0185a54e566, State: Initialized, Role: FOLLOWER
I20260926 20:56:18.765324 6531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } }
I20260926 20:56:18.765420 6531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:18.765455 6531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:18.765527 6531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:18.766320 6531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } }
I20260926 20:56:18.766433 6531 leader_election.cc:302] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 896ea7692253410faae3e0185a54e566; no voters:
I20260926 20:56:18.766631 6531 leader_election.cc:288] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:18.766731 6536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:18.766903 6536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [term 1 LEADER]: Becoming Leader. State: Replica: 896ea7692253410faae3e0185a54e566, State: Running, Role: LEADER
I20260926 20:56:18.766911 6531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:18.767026 6536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } }
I20260926 20:56:18.767366 6537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } }
I20260926 20:56:18.767478 6537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:18.767665 6538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 896ea7692253410faae3e0185a54e566. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } }
I20260926 20:56:18.767740 6538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:18.767859 6543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:18.768437 6543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:18.768957 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:18.758003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.0.98.62:44429)
0926 20:56:18.758186 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:18.758194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:18.758220 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:56:18.758654 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:18.758661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:18.758710 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:18.758828 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:18.759614 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:18.760590 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:18.761608 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:18.762025 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:18.762129 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:18.764605 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:18.764626 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:18.764627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:18.764642 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:18.766785 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:18.767226 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:18.767229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:18.767230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:18.767329 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:18.767547 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:18.767548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:18.767548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:18.767889 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:18.768051 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:18.768704 (+ 653us) server_negotiation.cc:299] Negotiation successful
0926 20:56:18.768812 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260926 20:56:18.770581 6543 catalog_manager.cc:1347] Generated new cluster ID: 8e199d17d12345d1868af6806326bb00
I20260926 20:56:18.770651 6543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:56:18.801709 6543 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:18.802299 6543 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:18.806212 6543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566: Generated new TSK 0
I20260926 20:56:18.806419 6543 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 20:56:18.852706 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44429
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:18.949156 6557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:18.949397 6557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:18.949441 6557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:18.950655 6557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:18.950719 6557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:18.950769 6557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:56:18.952280 6557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:18.952579 6557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:18.952823 6557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:18.958132 6563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:18.958122 6562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:18.958134 6565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:18.958161 6557 server_base.cc:1034] running on GCE node
I20260926 20:56:18.958562 6557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:18.958829 6557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:18.960008 6557 hybrid_clock.cc:648] HybridClock initialized: now 1790456178959977 us; error 50 us; skew 500 ppm
Sep 26 20:56:18 dist-test-slave-fl7g krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456178, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:18.962814 6557 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 20:56:18.963230 6557 webserver.cc:460] Webserver started at http://127.0.98.1:42191/ using document root <none> and password file <none>
I20260926 20:56:18.963418 6557 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:18.963459 6557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:18.963558 6557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:18.964675 6557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "a812960f384e4f578bbf8392b67794c5"
format_stamp: "Formatted at 2026-09-26 20:56:18 on dist-test-slave-fl7g"
server_key: "fe8a6fdf23f1eab8b7049ec513b979f1"
server_key_iv: "9dbb788cbda656a3504ffbb04c05292e"
server_key_version: "encryptionkey@0"
I20260926 20:56:18.964999 6557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a812960f384e4f578bbf8392b67794c5"
format_stamp: "Formatted at 2026-09-26 20:56:18 on dist-test-slave-fl7g"
server_key: "fe8a6fdf23f1eab8b7049ec513b979f1"
server_key_iv: "9dbb788cbda656a3504ffbb04c05292e"
server_key_version: "encryptionkey@0"
I20260926 20:56:18.966306 6557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:18.967144 6572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:18.967306 6557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:18.967401 6557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "a812960f384e4f578bbf8392b67794c5"
format_stamp: "Formatted at 2026-09-26 20:56:18 on dist-test-slave-fl7g"
server_key: "fe8a6fdf23f1eab8b7049ec513b979f1"
server_key_iv: "9dbb788cbda656a3504ffbb04c05292e"
server_key_version: "encryptionkey@0"
I20260926 20:56:18.967492 6557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:18.978315 6557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:18.980727 6557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:18.980893 6557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:18.981148 6557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:18.981463 6557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:18.981516 6557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:18.981556 6557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:18.981596 6557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:18.987321 6557 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42067
I20260926 20:56:18.987361 6685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42067 every 8 connection(s)
I20260926 20:56:18.987681 6557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:56:18.988514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 6557
I20260926 20:56:18.988670 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:56:18.988919 392 external_mini_cluster.cc:1428] Setting key d4a045f509dbc0929d2eb4ef399353db
Sep 26 20:56:18 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456178, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 20:56:18.998391 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:18.988530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60243 (local address 127.0.98.62:44429)
0926 20:56:18.988596 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:18.988599 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:18.988906 (+ 307us) server_negotiation.cc:407] Connection header received
0926 20:56:18.989580 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:18.989580 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:18.989593 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:18.989664 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:18.991055 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:18.991576 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:18.992687 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:18.992897 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:18.992980 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:18.995200 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:18.995202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:18.995202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:18.995211 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:18.996836 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:18.997365 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:18.997367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:18.997367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:18.997428 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:18.997705 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:18.997706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:18.997707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:18.997832 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:18.997908 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:18.998239 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 20:56:18.998320 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:56:18.998566 6688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:18.988529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.1:60243)
0926 20:56:18.988727 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:18.988781 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:18.989450 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:18.989744 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:18.989749 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:18.990625 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:18.990913 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:18.990920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:18.991683 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:18.991685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:18.992584 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:18.992586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:18.992980 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:18.992981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:18.993140 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:18.993722 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:18.993732 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:18.995101 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:18.996959 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:18.996966 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:18.996969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:18.997277 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:18.997500 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:18.997501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:18.997501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:18.997638 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:18.997898 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:18.997905 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:18.998063 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 20:56:18.998191 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 20:56:18.998351 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":86,"threads_started":1}
I20260926 20:56:18.999086 6686 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44429
I20260926 20:56:18.999213 6686 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:18.999398 6686 heartbeater.cc:511] Master 127.0.98.62:44429 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260926 20:56:19.000015 6495 ts_manager.cc:194] Registered new tserver with Master: a812960f384e4f578bbf8392b67794c5 (127.0.98.1:42067)
I20260926 20:56:19.001024 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 20:56:19.016083 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44429
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:19.098698 6691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:19.098933 6691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:19.098960 6691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:19.100229 6691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:19.100286 6691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:19.100314 6691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:56:19.101920 6691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:19.102211 6691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:19.102476 6691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:19.107838 6697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:19.107836 6696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:19.107836 6699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:19.108177 6691 server_base.cc:1034] running on GCE node
I20260926 20:56:19.108453 6691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:19.108731 6691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:19.109900 6691 hybrid_clock.cc:648] HybridClock initialized: now 1790456179109879 us; error 32 us; skew 500 ppm
Sep 26 20:56:19 dist-test-slave-fl7g krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456179, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:19.112648 6691 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 20:56:19.113088 6691 webserver.cc:460] Webserver started at http://127.0.98.2:33245/ using document root <none> and password file <none>
I20260926 20:56:19.113351 6691 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:19.113399 6691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:19.113560 6691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:19.114689 6691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "2cdb882ced5c4fc383c0ade935a01cab"
format_stamp: "Formatted at 2026-09-26 20:56:19 on dist-test-slave-fl7g"
server_key: "5bf80a061563aacd01d41c6f20d8acfb"
server_key_iv: "801aeec36936670675a5f193cdea943a"
server_key_version: "encryptionkey@0"
I20260926 20:56:19.115067 6691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2cdb882ced5c4fc383c0ade935a01cab"
format_stamp: "Formatted at 2026-09-26 20:56:19 on dist-test-slave-fl7g"
server_key: "5bf80a061563aacd01d41c6f20d8acfb"
server_key_iv: "801aeec36936670675a5f193cdea943a"
server_key_version: "encryptionkey@0"
I20260926 20:56:19.116504 6691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 20:56:19.117317 6706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:19.117481 6691 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:19.117578 6691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "2cdb882ced5c4fc383c0ade935a01cab"
format_stamp: "Formatted at 2026-09-26 20:56:19 on dist-test-slave-fl7g"
server_key: "5bf80a061563aacd01d41c6f20d8acfb"
server_key_iv: "801aeec36936670675a5f193cdea943a"
server_key_version: "encryptionkey@0"
I20260926 20:56:19.117698 6691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:19.124291 6691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:19.126145 6691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:19.126313 6691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:19.126549 6691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:19.126861 6691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:19.126911 6691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:19.126955 6691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:19.126982 6691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:19.132861 6691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46125
I20260926 20:56:19.132902 6819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46125 every 8 connection(s)
I20260926 20:56:19.133250 6691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:56:19 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456179, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:56:19.141196 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 6691
I20260926 20:56:19.141328 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:56:19.141574 392 external_mini_cluster.cc:1428] Setting key 71d2202c3f4980e72bfe36450af286d1
I20260926 20:56:19.144290 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.134062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54251 (local address 127.0.98.62:44429)
0926 20:56:19.134134 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.134136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.134382 (+ 246us) server_negotiation.cc:407] Connection header received
0926 20:56:19.135122 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.135124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.135153 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.135215 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.136243 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.136817 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.137956 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.138179 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.138261 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.140850 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.140852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.140852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.140860 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.142833 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.143269 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.143270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.143270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.143334 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.143601 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.143602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.143603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.143735 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.143806 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.144110 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.144195 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:56:19.144773 6822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.134044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.2:54251)
0926 20:56:19.134196 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:19.134241 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:19.135006 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:19.135293 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:19.135297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:19.135909 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:19.136119 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.136123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.136920 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.136921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.137863 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.137866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.138267 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.138268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.138412 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.138969 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:19.138978 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:19.140707 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:19.142923 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.142933 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.142935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.143190 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.143401 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.143402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.143402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.143520 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.143810 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:19.143817 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:19.143963 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 20:56:19.144073 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 20:56:19.144253 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260926 20:56:19.145308 6820 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44429
I20260926 20:56:19.145416 6820 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:19.145637 6820 heartbeater.cc:511] Master 127.0.98.62:44429 requested a full tablet report, sending...
I20260926 20:56:19.146087 6495 ts_manager.cc:194] Registered new tserver with Master: 2cdb882ced5c4fc383c0ade935a01cab (127.0.98.2:46125)
I20260926 20:56:19.146520 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54251
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 20:56:19.168292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44429
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:19.250872 6825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:19.251112 6825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:19.251138 6825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:19.252372 6825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:19.252470 6825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:19.252519 6825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:56:19.254165 6825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:19.254442 6825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:19.254693 6825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:19.259820 6833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:19.259836 6830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:19.259837 6831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:19.259922 6825 server_base.cc:1034] running on GCE node
I20260926 20:56:19.260227 6825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:19.260517 6825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:19.261660 6825 hybrid_clock.cc:648] HybridClock initialized: now 1790456179261647 us; error 28 us; skew 500 ppm
Sep 26 20:56:19 dist-test-slave-fl7g krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456179, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:19.264169 6825 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 20:56:19.264678 6825 webserver.cc:460] Webserver started at http://127.0.98.3:39359/ using document root <none> and password file <none>
I20260926 20:56:19.264881 6825 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:19.264923 6825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:19.265015 6825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:19.266037 6825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "7b23b75bbf57466498a372ea17df29bf"
format_stamp: "Formatted at 2026-09-26 20:56:19 on dist-test-slave-fl7g"
server_key: "d71b70502be78506296e5723a1c6c6ff"
server_key_iv: "0913f004b85177bed916d1a0a15de83b"
server_key_version: "encryptionkey@0"
I20260926 20:56:19.266350 6825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7b23b75bbf57466498a372ea17df29bf"
format_stamp: "Formatted at 2026-09-26 20:56:19 on dist-test-slave-fl7g"
server_key: "d71b70502be78506296e5723a1c6c6ff"
server_key_iv: "0913f004b85177bed916d1a0a15de83b"
server_key_version: "encryptionkey@0"
I20260926 20:56:19.267998 6825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:56:19.269014 6840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:19.269309 6825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:56:19.269398 6825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "7b23b75bbf57466498a372ea17df29bf"
format_stamp: "Formatted at 2026-09-26 20:56:19 on dist-test-slave-fl7g"
server_key: "d71b70502be78506296e5723a1c6c6ff"
server_key_iv: "0913f004b85177bed916d1a0a15de83b"
server_key_version: "encryptionkey@0"
I20260926 20:56:19.269466 6825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:19.289049 6825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:19.291167 6825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:19.291285 6825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:19.291481 6825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:19.291818 6825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:19.291854 6825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:19.291898 6825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:19.291925 6825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:19.299873 6825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41389
I20260926 20:56:19.299902 6953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41389 every 8 connection(s)
I20260926 20:56:19.300283 6825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:56:19.303689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 6825
I20260926 20:56:19.303896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:56:19.304148 392 external_mini_cluster.cc:1428] Setting key fd315a7a01cdaf2c03447d098bececd5
Sep 26 20:56:19 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456179, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:56:19.312600 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.301184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59799 (local address 127.0.98.62:44429)
0926 20:56:19.301234 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.301237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.301759 (+ 522us) server_negotiation.cc:407] Connection header received
0926 20:56:19.302365 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.302367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.302379 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.302445 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.303551 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.304273 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.305800 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.306059 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.306130 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.308817 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.308820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.308820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.308828 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.310722 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.311311 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.311312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.311313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.311386 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.311729 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.311731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.311731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.311894 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.311971 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.312343 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.312482 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:56:19.312903 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.301187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.3:59799)
0926 20:56:19.301375 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:19.301442 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:19.302233 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:19.302614 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:19.302618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:19.303208 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:19.303415 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.303420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.304413 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.304444 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.305717 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.305720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.306147 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.306148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.306321 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.307056 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:19.307066 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:19.308714 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:19.310827 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.310835 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.310837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.311222 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.311485 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.311486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.311486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.311651 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.311970 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:19.311975 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:19.312148 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 20:56:19.312298 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 20:56:19.312620 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260926 20:56:19.313632 6954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44429
I20260926 20:56:19.313762 6954 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:19.313985 6954 heartbeater.cc:511] Master 127.0.98.62:44429 requested a full tablet report, sending...
I20260926 20:56:19.314467 6495 ts_manager.cc:194] Registered new tserver with Master: 7b23b75bbf57466498a372ea17df29bf (127.0.98.3:41389)
I20260926 20:56:19.314963 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59799
I20260926 20:56:19.316375 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260926 20:56:19.348376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40951
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44429,127.0.98.61:40951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:19.431389 6959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:19.431630 6959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:19.431654 6959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:19.432950 6959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:19.432997 6959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:19.433012 6959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:19.433025 6959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:19.433038 6959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:19.434561 6959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44429,127.0.98.61:40951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
W20260926 20:56:19.434701 6959 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:44429,127.0.98.61:40951'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 20:56:19.434860 6959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:19.435099 6959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:19.440166 6965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:19.440152 6964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:19.440155 6967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:19.440480 6959 server_base.cc:1034] running on GCE node
I20260926 20:56:19.440754 6959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:19.441097 6959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:19.442303 6959 hybrid_clock.cc:648] HybridClock initialized: now 1790456179442289 us; error 44 us; skew 500 ppm
Sep 26 20:56:19 dist-test-slave-fl7g krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456179, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:19.445036 6959 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 20:56:19.445545 6959 webserver.cc:460] Webserver started at http://127.0.98.61:35889/ using document root <none> and password file <none>
I20260926 20:56:19.445746 6959 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:19.445789 6959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:19.445883 6959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:19.446935 6959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/data/instance:
uuid: "b30454144cd14defb1a3d00fb2cb347e"
format_stamp: "Formatted at 2026-09-26 20:56:19 on dist-test-slave-fl7g"
server_key: "81e395a201f51a23c003706484cb4459"
server_key_iv: "794475da180a8c348dcd08fb88513ab4"
server_key_version: "encryptionkey@0"
I20260926 20:56:19.447247 6959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/wal/instance:
uuid: "b30454144cd14defb1a3d00fb2cb347e"
format_stamp: "Formatted at 2026-09-26 20:56:19 on dist-test-slave-fl7g"
server_key: "81e395a201f51a23c003706484cb4459"
server_key_iv: "794475da180a8c348dcd08fb88513ab4"
server_key_version: "encryptionkey@0"
I20260926 20:56:19.448676 6959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:56:19.449487 6974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:19.449666 6959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:19.449769 6959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/wal
uuid: "b30454144cd14defb1a3d00fb2cb347e"
format_stamp: "Formatted at 2026-09-26 20:56:19 on dist-test-slave-fl7g"
server_key: "81e395a201f51a23c003706484cb4459"
server_key_iv: "794475da180a8c348dcd08fb88513ab4"
server_key_version: "encryptionkey@0"
I20260926 20:56:19.449849 6959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:19.479954 6959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:19.482168 6959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:19.482339 6959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:19.486495 6959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40951
I20260926 20:56:19.486550 7026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40951 every 8 connection(s)
I20260926 20:56:19.486929 6959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/data/info.pb
I20260926 20:56:19.487964 7027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:19.489744 7027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:56:19.493837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 6959
Sep 26 20:56:19 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456179, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 20:56:19 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 20:56:19.500170 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.490557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45962 (local address 127.0.98.62:44429)
0926 20:56:19.490622 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.490624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.490849 (+ 225us) server_negotiation.cc:407] Connection header received
0926 20:56:19.491335 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.491336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.491345 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.491408 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.492058 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.492599 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.493673 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.493980 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.494050 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.496799 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.496802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.496802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.496810 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.498537 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.499011 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.499013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.499013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.499083 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.499372 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.499373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.499373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.499536 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.499612 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.499992 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.500090 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:56:19.500497 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.490523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.0.1:45962)
0926 20:56:19.490714 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:19.490761 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:19.491254 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:19.491481 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:19.491485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:19.491761 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:19.491974 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.491978 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.492678 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.492680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.493576 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.493578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.494088 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.494089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.494205 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.495013 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:19.495025 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:19.496707 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:19.498663 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.498671 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.498674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.498928 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.499153 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.499154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.499155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.499276 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.499618 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:19.499625 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:19.499815 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 20:56:19.499950 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 20:56:19.500150 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":113,"threads_started":1}
I20260926 20:56:19.501286 7027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:56:19.504060 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.494406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41054 (local address 127.0.98.61:40951)
0926 20:56:19.494580 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.494584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.494600 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:56:19.494638 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.494640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.494656 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.494744 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.495324 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.496310 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.497244 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.497559 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.497638 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.500843 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.500847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.500848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.500855 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.502710 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.503125 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.503128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.503129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.503193 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.503403 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.503404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.503405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.503594 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.503769 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.503845 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.503932 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
Sep 26 20:56:19 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456179, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 20:56:19.508363 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.501573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40951 (local address 127.0.0.1:41060)
0926 20:56:19.501627 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:19.501639 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:19.501678 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:19.501908 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:19.501909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:19.502054 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:19.502099 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.502101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.502789 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.502790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.503325 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.503327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.503806 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.503807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.503875 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.504358 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:19.504363 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:19.505648 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:19.507473 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.507474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.507474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.507645 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.507857 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.507857 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.507857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.507909 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.508124 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:19.508125 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:19.508160 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 20:56:19.508215 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 20:56:19.508295 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 20:56:19.508546 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.501646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.0.98.61:40951)
0926 20:56:19.501780 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.501782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.501791 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:56:19.501816 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.501816 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.501826 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.501906 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.502183 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.502699 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.503416 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.503734 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.503812 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.505722 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.505723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.505723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.505730 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.507400 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.507729 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.507730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.507730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.507791 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.507960 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.507961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.507961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.508060 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.508109 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.508229 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.508344 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260926 20:56:19.509845 7027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: Bootstrap starting.
I20260926 20:56:19.510481 7027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:19.510768 7027 log.cc:824] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:19.511530 7027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: No bootstrap required, opened a new log
I20260926 20:56:19.513031 7027 raft_consensus.cc:348] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } }
I20260926 20:56:19.513163 7027 raft_consensus.cc:374] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:19.513188 7027 raft_consensus.cc:729] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b30454144cd14defb1a3d00fb2cb347e, State: Initialized, Role: FOLLOWER
I20260926 20:56:19.513317 7027 consensus_queue.cc:260] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } }
I20260926 20:56:19.513630 7034 sys_catalog.cc:455] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } } }
I20260926 20:56:19.513697 7027 sys_catalog.cc:564] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:19.513727 7034 sys_catalog.cc:458] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:56:19.515332 7045 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:56:19.515394 7045 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:56:19.517060 6959 master_runner.cc:428] Detected that this master b30454144cd14defb1a3d00fb2cb347e is joining an existing cluster
I20260926 20:56:19.517105 6959 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:40951
I20260926 20:56:19.524955 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.518019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.0.1:45976)
0926 20:56:19.518177 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:19.518187 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:19.518267 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:19.518423 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:19.518424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:19.518629 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:19.518713 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.518715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.519397 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.519398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.520303 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.520304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.520728 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.520729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.520792 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.521298 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:19.521303 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:19.521771 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:19.523603 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.523604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.523605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.523849 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.524045 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.524046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.524046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.524115 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.524584 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:19.524585 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:19.524642 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 20:56:19.524719 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:56:19.524842 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
I20260926 20:56:19.525348 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.518067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.0.98.62:44429)
0926 20:56:19.518119 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.518122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.518251 (+ 129us) server_negotiation.cc:407] Connection header received
0926 20:56:19.518330 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.518331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.518344 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.518404 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.518800 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.519319 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.520380 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.520638 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.520713 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.521915 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.521916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.521917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.521930 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.523502 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.523919 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.523920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.523921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.523982 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.524189 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.524190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.524191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.524326 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.524396 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.525174 (+ 778us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.525258 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 20:56:19.525207 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.518119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.0.98.61:40951)
0926 20:56:19.518156 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.518158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.518410 (+ 252us) server_negotiation.cc:407] Connection header received
0926 20:56:19.518476 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.518477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.518486 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.518539 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.518858 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.519604 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.520973 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.521317 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.521383 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.522383 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.522384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.522384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.522389 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.524058 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.524395 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.524396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.524397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.524494 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.524733 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.524734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.524734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.524838 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.524906 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.525048 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.525128 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:56:19.525357 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.518185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40951 (local address 127.0.0.1:41066)
0926 20:56:19.518361 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:19.518370 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:19.518430 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:19.518544 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:19.518545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:19.518706 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:19.518751 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.518752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.519685 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.519686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.520864 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.520866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.521443 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.521443 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.521497 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.521915 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:19.521920 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:19.522292 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:19.524135 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.524136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.524136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.524328 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.524565 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.524565 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.524566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.524653 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.524898 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:19.524899 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:19.524949 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 20:56:19.525026 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:56:19.525299 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
Sep 26 20:56:19 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456178, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 20:56:19.535450 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.527790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.0.98.61:40951)
0926 20:56:19.527833 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.527834 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.527999 (+ 165us) server_negotiation.cc:407] Connection header received
0926 20:56:19.528083 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.528084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.528093 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.528142 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.528682 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.529130 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.530268 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.530438 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.530492 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.532793 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.532794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.532794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.532806 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.534198 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.534645 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.534646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.534647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.534703 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.534898 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.534899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.534899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.535014 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.535087 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.535269 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.535349 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:56:19.535492 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.527715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40951 (local address 127.0.0.1:41070)
0926 20:56:19.527868 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:19.527918 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:19.528007 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:19.528161 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:19.528165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:19.528542 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:19.528609 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.528612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.529206 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.529207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.530178 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.530181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.530508 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.530509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.530601 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.531135 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:19.531140 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:19.532690 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:19.534302 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.534311 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.534314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.534557 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.534775 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.534776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.534776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.534839 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.535081 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:19.535090 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:19.535147 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 20:56:19.535243 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 20:56:19.535368 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260926 20:56:19.536280 6495 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:40951: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 40951
}
attrs {
promote: true
}
}
dest_uuid: "896ea7692253410faae3e0185a54e566"
cas_config_opid_index: -1
I20260926 20:56:19.536638 6495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } }
I20260926 20:56:19.537112 7055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b30454144cd14defb1a3d00fb2cb347e (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } } }
I20260926 20:56:19.537462 7001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:19.538026 7055 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:40951
I20260926 20:56:19.538280 6959 master.cc:537] Master@127.0.98.61:40951 shutting down...
I20260926 20:56:19.538415 7001 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 1 FOLLOWER]: Refusing update from remote peer 896ea7692253410faae3e0185a54e566: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 20:56:19.538755 7055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:56:19.539498 6482 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 20:56:19.540045 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } } }
I20260926 20:56:19.540138 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: This master's current role is: LEADER
W20260926 20:56:19.540462 6482 consensus_peers.cc:487] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 -> Peer b30454144cd14defb1a3d00fb2cb347e (127.0.98.61:40951): Couldn't send request to peer b30454144cd14defb1a3d00fb2cb347e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:56:19.541359 6959 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:19.541468 6959 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:19.541520 6959 tablet_replica.cc:330] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: stopping tablet replica
I20260926 20:56:19.554236 6959 master.cc:552] Master@127.0.98.61:40951 shutdown complete.
I20260926 20:56:19.554296 6959 master_runner.cc:305] Clearing existing system tablet
I20260926 20:56:19.554448 6959 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:19.554497 6959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:19.555210 6959 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:19.555794 7059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:19.555931 6959 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:19.555992 6959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:19.556016 6959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/wal
uuid: "b30454144cd14defb1a3d00fb2cb347e"
format_stamp: "Formatted at 2026-09-26 20:56:19 on dist-test-slave-fl7g"
server_key: "81e395a201f51a23c003706484cb4459"
server_key_iv: "794475da180a8c348dcd08fb88513ab4"
server_key_version: "encryptionkey@0"
I20260926 20:56:19.557222 6959 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:19.558840 6959 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 20:56:19.558905 6959 log.cc:1196] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 20:56:19.559118 6959 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: Deleting consensus metadata
I20260926 20:56:19.559258 6959 master_runner.cc:315] Copying system tablet from 127.0.98.62:44429
I20260926 20:56:19.559921 6959 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:44429
I20260926 20:56:19.567027 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.560338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.0.98.62:44429)
0926 20:56:19.560395 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.560398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.560571 (+ 173us) server_negotiation.cc:407] Connection header received
0926 20:56:19.560654 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.560656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.560666 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.560722 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.561094 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.561677 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.562971 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.563253 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.563338 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.564366 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.564368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.564368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.564378 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.565782 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.566174 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.566175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.566176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.566239 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.566445 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.566446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.566447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.566610 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.566678 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.566838 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.566921 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 20:56:19.567058 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.560284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.0.1:45982)
0926 20:56:19.560494 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:19.560507 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:19.560588 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:19.560742 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:19.560743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:19.560943 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:19.561021 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.561023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.561796 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.561798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.562879 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.562882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.563337 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.563338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.563411 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.563922 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:19.563927 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:19.564273 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:19.565888 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.565889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.565890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.566085 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.566307 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.566308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.566309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.566372 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.566668 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:19.566669 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:19.566718 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 20:56:19.566809 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 20:56:19.566922 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260926 20:56:19.567523 6515 tablet_copy_service.cc:140] P 896ea7692253410faae3e0185a54e566: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b30454144cd14defb1a3d00fb2cb347e ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:45982)
I20260926 20:56:19.567641 6515 tablet_copy_service.cc:161] P 896ea7692253410faae3e0185a54e566: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b30454144cd14defb1a3d00fb2cb347e at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:45982: session id = b30454144cd14defb1a3d00fb2cb347e-00000000000000000000000000000000
I20260926 20:56:19.568367 6515 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 20:56:19.569058 6959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 20:56:19.570079 6959 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: tablet copy: Starting download of 0 data blocks...
I20260926 20:56:19.570279 6959 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: tablet copy: Starting download of 1 WAL segments...
I20260926 20:56:19.571317 6959 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 20:56:19.572273 6515 tablet_copy_service.cc:342] P 896ea7692253410faae3e0185a54e566: Request end of tablet copy session b30454144cd14defb1a3d00fb2cb347e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:45982
I20260926 20:56:19.572341 6515 tablet_copy_service.cc:433] P 896ea7692253410faae3e0185a54e566: ending tablet copy session b30454144cd14defb1a3d00fb2cb347e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b30454144cd14defb1a3d00fb2cb347e
I20260926 20:56:19.575215 6959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:19.576411 7067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:19.576495 7066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:19.581585 7069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:19.581593 6959 server_base.cc:1034] running on GCE node
I20260926 20:56:19.581756 6959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:19.581893 6959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:19.583026 6959 hybrid_clock.cc:648] HybridClock initialized: now 1790456179583019 us; error 41 us; skew 500 ppm
Sep 26 20:56:19 dist-test-slave-fl7g krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456179, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:19.584547 6959 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 20:56:19.584846 6959 webserver.cc:460] Webserver started at http://127.0.98.61:40843/ using document root <none> and password file <none>
I20260926 20:56:19.584987 6959 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:19.585037 6959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:19.585680 6959 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:56:19.586227 7076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:19.586344 6959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:56:19.586390 6959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/wal
uuid: "b30454144cd14defb1a3d00fb2cb347e"
format_stamp: "Formatted at 2026-09-26 20:56:19 on dist-test-slave-fl7g"
server_key: "81e395a201f51a23c003706484cb4459"
server_key_iv: "794475da180a8c348dcd08fb88513ab4"
server_key_version: "encryptionkey@0"
I20260926 20:56:19.586449 6959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:19.607124 6959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:19.608737 6959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:19.611799 6959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40951
I20260926 20:56:19.611819 7128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40951 every 8 connection(s)
I20260926 20:56:19.612145 6959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-1/data/info.pb
I20260926 20:56:19.613077 7129 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:56:19.613636 7129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: Bootstrap starting.
I20260926 20:56:19.616743 7129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:56:19.616979 7129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: Bootstrap complete.
I20260926 20:56:19.617172 7129 raft_consensus.cc:348] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } }
I20260926 20:56:19.617260 7129 raft_consensus.cc:729] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b30454144cd14defb1a3d00fb2cb347e, State: Initialized, Role: LEARNER
I20260926 20:56:19.617298 7129 consensus_queue.cc:260] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } }
I20260926 20:56:19.617477 7129 sys_catalog.cc:564] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:19.617496 7132 sys_catalog.cc:455] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } } }
I20260926 20:56:19.617601 7132 sys_catalog.cc:458] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [sys.catalog]: This master's current role is: LEARNER
I20260926 20:56:19.618867 7143 catalog_manager.cc:1259] Loaded cluster ID: 8e199d17d12345d1868af6806326bb00
I20260926 20:56:19.618920 7143 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: loading cluster ID for follower catalog manager: success
I20260926 20:56:19.619877 7143 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: acquiring CA information for follower catalog manager: success
I20260926 20:56:19.620167 7143 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:56:19.662629 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.656803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.0.98.61:40951)
0926 20:56:19.656927 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.656929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.656938 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:56:19.656963 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.656964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.656980 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.657052 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.657574 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.658201 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.658862 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.659044 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.659092 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.660184 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.660186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.660186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.660193 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.661642 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.661914 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.661915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.661915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.661964 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.662141 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.662141 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.662141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.662234 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.662301 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.662470 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.662531 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260926 20:56:19.687577 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35517
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44429,127.0.98.61:40951,127.0.98.60:35517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:19.775983 7147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:19.776222 7147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:19.776244 7147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:19.777374 7147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:19.777411 7147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:19.777426 7147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:19.777437 7147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:19.777449 7147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:19.778913 7147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44429,127.0.98.61:40951,127.0.98.60:35517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:19.779114 7147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:19.779285 7147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:19.784129 7153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:19.784191 7152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:19.784271 7155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:19.784873 7147 server_base.cc:1034] running on GCE node
I20260926 20:56:19.785167 7147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:19.785451 7147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:19.786583 7147 hybrid_clock.cc:648] HybridClock initialized: now 1790456179786569 us; error 27 us; skew 500 ppm
Sep 26 20:56:19 dist-test-slave-fl7g krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456179, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:19.789237 7147 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 20:56:19.789763 7147 webserver.cc:460] Webserver started at http://127.0.98.60:44719/ using document root <none> and password file <none>
I20260926 20:56:19.789987 7147 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:19.790035 7147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:19.790190 7147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:19.791294 7147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/data/instance:
uuid: "79c1cf7b052a412db5bb1a341b1b90d0"
format_stamp: "Formatted at 2026-09-26 20:56:19 on dist-test-slave-fl7g"
server_key: "b3c7759bf9dd3cdd8211c5f156f55e93"
server_key_iv: "b3a8f9db909c6cfc35bf92b3ce1579ca"
server_key_version: "encryptionkey@0"
I20260926 20:56:19.791673 7147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/wal/instance:
uuid: "79c1cf7b052a412db5bb1a341b1b90d0"
format_stamp: "Formatted at 2026-09-26 20:56:19 on dist-test-slave-fl7g"
server_key: "b3c7759bf9dd3cdd8211c5f156f55e93"
server_key_iv: "b3a8f9db909c6cfc35bf92b3ce1579ca"
server_key_version: "encryptionkey@0"
I20260926 20:56:19.793123 7147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:56:19.793924 7162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:19.794114 7147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:19.794202 7147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/wal
uuid: "79c1cf7b052a412db5bb1a341b1b90d0"
format_stamp: "Formatted at 2026-09-26 20:56:19 on dist-test-slave-fl7g"
server_key: "b3c7759bf9dd3cdd8211c5f156f55e93"
server_key_iv: "b3a8f9db909c6cfc35bf92b3ce1579ca"
server_key_version: "encryptionkey@0"
I20260926 20:56:19.794272 7147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:19.803521 7147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:19.805871 7147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:19.806015 7147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:19.810964 7147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35517
I20260926 20:56:19.811040 7214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35517 every 8 connection(s)
I20260926 20:56:19.811376 7147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/data/info.pb
I20260926 20:56:19.812361 7215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:19.812930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 7147
I20260926 20:56:19.814409 7215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:56:19 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456179, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 20:56:19 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 20:56:19.823594 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.815169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45992 (local address 127.0.98.62:44429)
0926 20:56:19.815222 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.815224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.815322 (+ 98us) server_negotiation.cc:407] Connection header received
0926 20:56:19.815392 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.815393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.815402 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.815446 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.816037 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.816615 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.817627 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.817879 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.817942 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.820341 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.820342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.820343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.820349 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.822045 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.822487 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.822489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.822489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.822550 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.822854 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.822855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.822856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.823006 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.823091 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.823425 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.823525 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 20:56:19.823964 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.815057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.0.1:45992)
0926 20:56:19.815225 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:19.815269 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:19.815334 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:19.815461 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:19.815464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:19.815814 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:19.815919 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.815924 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.816699 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.816700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.817556 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.817558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.817944 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.817945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.818012 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.818614 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:19.818622 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:19.820239 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:19.822137 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.822143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.822145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.822396 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.822619 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.822620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.822620 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.822773 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.823078 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:19.823097 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:19.823268 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 20:56:19.823363 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 20:56:19.823594 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":25,"thread_start_us":110,"threads_started":1}
I20260926 20:56:19.823964 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.813607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.0.98.60:35517)
0926 20:56:19.813803 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.813812 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.813841 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:56:19.814258 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.814264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.814320 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.814493 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.815321 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.816525 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.817254 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.817798 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.817896 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.820924 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.820931 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.820932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.820939 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.822705 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.823123 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.823125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.823126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.823180 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.823412 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.823413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.823413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.823562 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.823701 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.823808 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.823877 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260926 20:56:19.824692 7215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:56:19 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456179, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 20:56:19.831202 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.824962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40951 (local address 127.0.0.1:41102)
0926 20:56:19.824998 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:19.825010 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:19.825050 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:19.825164 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:19.825164 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:19.825269 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:19.825313 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.825314 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.825899 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.825899 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.826392 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.826393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.826742 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.826743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.826796 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.827225 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:19.827230 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:19.828579 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:19.830153 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.830154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.830154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.830333 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.830553 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.830553 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.830554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.830614 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.830861 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:19.830862 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:19.830908 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 20:56:19.830990 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:56:19.831099 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 20:56:19.831235 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.825026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.0.98.61:40951)
0926 20:56:19.825071 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.825073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.825081 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:56:19.825105 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.825105 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.825117 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.825161 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.825365 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.825839 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.826477 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.826677 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.826734 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.828695 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.828697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.828697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.828705 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.830053 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.830414 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.830415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.830416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.830477 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.830672 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.830673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.830673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.830795 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.830870 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.831017 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.831105 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:56:19.831717 7215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:56:19 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456179, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 20:56:19.838152 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.832020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.0.98.60:35517)
0926 20:56:19.832072 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.832074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.832097 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:56:19.832156 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.832156 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.832165 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.832203 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.832501 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.832962 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.833619 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.833842 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.833912 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.835688 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.835689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.835690 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.835696 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.837072 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.837509 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.837510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.837510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.837561 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.837747 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.837748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.837748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.837852 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.837901 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.838015 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.838080 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:56:19.838335 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.831967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35517 (local address 127.0.0.1:37502)
0926 20:56:19.832063 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:19.832071 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:19.832108 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:19.832231 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:19.832232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:19.832345 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:19.832381 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.832383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.833041 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.833042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.833526 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.833528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.833925 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.833926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.833962 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.834391 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:19.834395 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:19.835598 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:19.837223 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.837224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.837224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.837406 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.837636 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.837637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.837637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.837696 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.837909 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:19.837910 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:19.837946 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 20:56:19.838001 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 20:56:19.838138 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":59}
I20260926 20:56:19.839751 7215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: Bootstrap starting.
I20260926 20:56:19.840665 7215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:19.840950 7215 log.cc:824] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:19.841657 7215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: No bootstrap required, opened a new log
I20260926 20:56:19.843185 7215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } }
I20260926 20:56:19.843315 7215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:19.843338 7215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c1cf7b052a412db5bb1a341b1b90d0, State: Initialized, Role: FOLLOWER
I20260926 20:56:19.843499 7215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } }
I20260926 20:56:19.843778 7220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } } }
I20260926 20:56:19.843830 7215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:19.843884 7220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:56:19.845547 7231 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:56:19.845608 7231 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:56:19.847791 7147 master_runner.cc:428] Detected that this master 79c1cf7b052a412db5bb1a341b1b90d0 is joining an existing cluster
I20260926 20:56:19.847848 7147 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:35517
I20260926 20:56:19.872617 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.849207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37516 (local address 127.0.98.60:35517)
0926 20:56:19.849316 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.849318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.849325 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:56:19.849343 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.849344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.849353 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.849408 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.849919 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.850627 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.851814 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.852115 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.852157 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.853400 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.853401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.853401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.853409 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.870723 (+ 17314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.871757 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.871759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.871759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.871823 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.872021 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.872022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.872023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.872151 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.872227 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.872364 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.872486 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":94}
I20260926 20:56:19.872689 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.849046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35517 (local address 127.0.0.1:37516)
0926 20:56:19.849257 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:19.849265 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:19.849305 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:19.849413 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:19.849414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:19.849760 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:19.849837 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.849839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.850712 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.850713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.851708 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.851712 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.852248 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.852249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.852314 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.852910 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:19.852914 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:19.853279 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:19.870819 (+ 17540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.870820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.870820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.871016 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.871894 (+ 878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.871895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.871895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.871953 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.872218 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:19.872219 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:19.872259 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 20:56:19.872342 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 20:56:19.872536 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":142,"thread_start_us":83,"threads_started":1}
I20260926 20:56:19.873858 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.848894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.0.98.61:40951)
0926 20:56:19.848980 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.848981 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.849159 (+ 178us) server_negotiation.cc:407] Connection header received
0926 20:56:19.849255 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.849256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.849266 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.849296 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.849817 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.850587 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.852207 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.852879 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.852938 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.870558 (+ 17620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.870559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.870560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.870569 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.872569 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.872968 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.872970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.872970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.873043 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.873250 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.873251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.873251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.873377 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.873417 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.873683 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.873770 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:56:19.874011 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.848887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.0.98.62:44429)
0926 20:56:19.848938 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.848940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.849111 (+ 171us) server_negotiation.cc:407] Connection header received
0926 20:56:19.849217 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.849218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.849232 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.849268 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.849929 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.850665 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.851824 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.852118 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.852157 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.870051 (+ 17894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.870054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.870054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.870064 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.871665 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.873166 (+ 1501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.873167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.873168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.873230 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.873425 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.873425 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.873426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.873561 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.873604 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.873829 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.873913 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:56:19.874013 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.848841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.0.1:45998)
0926 20:56:19.849047 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:19.849060 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:19.849139 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:19.849461 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:19.849461 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:19.849761 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:19.849837 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.849839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.850751 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.850752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.851708 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.851711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.852250 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.852250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.852314 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.869064 (+ 16750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:19.869071 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:19.869856 (+ 785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:19.871762 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.871763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.871763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.873078 (+ 1315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.873301 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.873302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.873302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.873357 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.873652 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:19.873652 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:19.873695 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 20:56:19.873782 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 20:56:19.873888 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":35,"thread_start_us":76,"threads_started":1}
I20260926 20:56:19.874373 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.848954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40951 (local address 127.0.0.1:41116)
0926 20:56:19.849086 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:19.849096 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:19.849185 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:19.849360 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:19.849360 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:19.849645 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:19.849724 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.849727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.850685 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.850686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.852092 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.852095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.853031 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.853032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.853099 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.853551 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:19.853556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:19.870200 (+ 16644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:19.872677 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.872678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.872679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.872889 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.873127 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.873128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.873128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.873186 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.873462 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:19.873463 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:19.873559 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 20:56:19.873990 (+ 431us) client_negotiation.cc:240] Negotiation successful
0926 20:56:19.874227 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260926 20:56:19.875383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 6557
I20260926 20:56:19.881927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42067
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44429
--webserver_port=42191
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44429,127.0.98.61:40951
--tserver_master_addrs=127.0.98.62:44429,127.0.98.61:40951,127.0.98.60:35517 with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:56:19 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456178, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 20:56:19.890038 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.881465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35517 (local address 127.0.0.1:37528)
0926 20:56:19.881517 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:19.881530 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:19.881569 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:19.881833 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:19.881834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:19.882052 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:19.882134 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.882137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.882851 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.882852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.883645 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.883648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.884240 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.884241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.884317 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.884865 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:19.884871 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:19.886444 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:19.888785 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.888787 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.888788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.889038 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.889273 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.889274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.889274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.889341 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.889668 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:19.889669 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:19.889723 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 20:56:19.889800 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:56:19.889920 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 20:56:19.890065 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.881650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37528 (local address 127.0.98.60:35517)
0926 20:56:19.881702 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.881703 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.881710 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:56:19.881754 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.881754 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.881763 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.881828 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.882225 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.882763 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.883745 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.884102 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.884177 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.886540 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.886542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.886543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.886553 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.888693 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.889125 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.889126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.889127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.889198 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.889413 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.889415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.889415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.889584 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.889674 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.889842 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.889932 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 20:56:19.891170 6495 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:35517: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 35517
}
attrs {
promote: true
}
}
dest_uuid: "896ea7692253410faae3e0185a54e566"
cas_config_opid_index: 5
I20260926 20:56:19.891458 6495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } }
I20260926 20:56:19.891871 7055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 79c1cf7b052a412db5bb1a341b1b90d0 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } } }
I20260926 20:56:19.892870 7055 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:35517
I20260926 20:56:19.892968 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } } }
I20260926 20:56:19.893041 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:19.893327 7189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:19.894270 7189 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 1 FOLLOWER]: Refusing update from remote peer 896ea7692253410faae3e0185a54e566: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 20:56:19.894577 7054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:19.896538 7189 raft_consensus.cc:858] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } }
W20260926 20:56:19.896607 7189 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } }
I20260926 20:56:19.897156 7189 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b30454144cd14defb1a3d00fb2cb347e (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } } }
I20260926 20:56:19.898111 7220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 896ea7692253410faae3e0185a54e566. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } } }
I20260926 20:56:19.898193 7220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 20:56:19.898221 7220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } } }
I20260926 20:56:19.898254 7220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 20:56:19.898277 7220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } } }
I20260926 20:56:19.898307 7220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 20:56:19.898389 7147 master.cc:537] Master@127.0.98.60:35517 shutting down...
I20260926 20:56:19.899369 7241 mvcc.cc:205] Tried to move back new op lower bound from 7333708508388663296 to 7333708508230234112. Current Snapshot: MvccSnapshot[applied={T|T < 7333708508239638528}]
I20260926 20:56:19.900794 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.894501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41126 (local address 127.0.98.61:40951)
0926 20:56:19.894548 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.894550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.895063 (+ 513us) server_negotiation.cc:407] Connection header received
0926 20:56:19.895164 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.895166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.895189 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.895232 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:19.895423 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.896212 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.897870 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.898509 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.898560 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.898638 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.900373 (+ 1735us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.900722 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:56:19.901054 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.892606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40951 (local address 127.0.0.1:41126)
0926 20:56:19.895011 (+ 2405us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:19.895023 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:19.895090 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:19.895309 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:19.895309 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:19.895311 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:19.895355 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.895356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.896351 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.896352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.897758 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.897761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.900174 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.900175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.900252 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.900278 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 20:56:19.900920 (+ 642us) client_negotiation.cc:240] Negotiation successful
0926 20:56:19.900982 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2387}
I20260926 20:56:19.901304 7103 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 1 LEARNER]: Refusing update from remote peer 896ea7692253410faae3e0185a54e566: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 20:56:19.901633 7054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 20:56:19.901762 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } } }
I20260926 20:56:19.901826 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:19.902431 7147 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260926 20:56:19.902518 7147 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260926 20:56:19.902549 7147 tablet_replica.cc:330] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: stopping tablet replica
I20260926 20:56:19.902688 7103 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 79c1cf7b052a412db5bb1a341b1b90d0 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } } }
I20260926 20:56:19.903795 7132 sys_catalog.cc:455] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } } }
I20260926 20:56:19.903887 7132 sys_catalog.cc:458] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [sys.catalog]: This master's current role is: LEARNER
I20260926 20:56:19.903821 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 896ea7692253410faae3e0185a54e566. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: true } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } } }
I20260926 20:56:19.904040 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [sys.catalog]: This master's current role is: LEARNER
W20260926 20:56:19.904102 6480 consensus_peers.cc:487] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 -> Peer 79c1cf7b052a412db5bb1a341b1b90d0 (127.0.98.60:35517): Couldn't send request to peer 79c1cf7b052a412db5bb1a341b1b90d0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:56:19.904703 7248 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566: attempting to promote NON_VOTER b30454144cd14defb1a3d00fb2cb347e to VOTER
I20260926 20:56:19.904918 7248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } }
I20260926 20:56:19.905565 7103 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 1 LEARNER]: Refusing update from remote peer 896ea7692253410faae3e0185a54e566: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 20:56:19.905834 7248 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260926 20:56:19.905978 6480 consensus_peers.cc:487] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 -> Peer 79c1cf7b052a412db5bb1a341b1b90d0 (127.0.98.60:35517): Couldn't send request to peer 79c1cf7b052a412db5bb1a341b1b90d0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:56:19.906620 7055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b30454144cd14defb1a3d00fb2cb347e (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } } }
I20260926 20:56:19.906759 7103 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b30454144cd14defb1a3d00fb2cb347e (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } } }
I20260926 20:56:19.907668 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } } }
I20260926 20:56:19.907742 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:19.907706 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } } }
I20260926 20:56:19.907785 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:19.908035 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } } }
I20260926 20:56:19.908118 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:56:19.921279 7147 master.cc:552] Master@127.0.98.60:35517 shutdown complete.
I20260926 20:56:19.921350 7147 master_runner.cc:305] Clearing existing system tablet
I20260926 20:56:19.921485 7147 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:19.921535 7147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:19.922284 7147 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.002s
I20260926 20:56:19.922860 7253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:19.923007 7147 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:19.923072 7147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:56:19.923096 7147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/wal
uuid: "79c1cf7b052a412db5bb1a341b1b90d0"
format_stamp: "Formatted at 2026-09-26 20:56:19 on dist-test-slave-fl7g"
server_key: "b3c7759bf9dd3cdd8211c5f156f55e93"
server_key_iv: "b3a8f9db909c6cfc35bf92b3ce1579ca"
server_key_version: "encryptionkey@0"
I20260926 20:56:19.923832 7147 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:19.925511 7147 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 20:56:19.925581 7147 log.cc:1196] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 20:56:19.925807 7147 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: Deleting consensus metadata
I20260926 20:56:19.925954 7147 master_runner.cc:315] Copying system tablet from 127.0.98.62:44429
I20260926 20:56:19.926553 7147 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:44429
I20260926 20:56:19.935178 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.926952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46010 (local address 127.0.98.62:44429)
0926 20:56:19.927000 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:19.927002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:19.927075 (+ 73us) server_negotiation.cc:407] Connection header received
0926 20:56:19.927151 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:19.927152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:19.927162 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:19.927239 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:19.927627 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.928394 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.929445 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.929750 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.929821 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.932106 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:19.932108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:19.932108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:19.932116 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:19.933873 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.934275 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.934276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.934277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.934338 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:19.934575 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:19.934577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:19.934577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:19.934750 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:19.934827 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:19.935002 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 20:56:19.935088 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:56:19.935174 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:19.926883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.0.1:46010)
0926 20:56:19.927012 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:19.927023 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:19.927092 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:19.927255 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:19.927256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:19.927465 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:19.927516 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.927519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.928523 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.928525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.929358 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:19.929361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:19.929837 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:19.929839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:19.929899 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:19.931546 (+ 1647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:19.931552 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:19.931989 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:19.933970 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.933971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.933972 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.934187 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.934413 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:19.934413 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:19.934414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:19.934473 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:19.934830 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:19.934831 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:19.934883 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 20:56:19.934964 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 20:56:19.935069 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260926 20:56:19.935907 6515 tablet_copy_service.cc:140] P 896ea7692253410faae3e0185a54e566: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 79c1cf7b052a412db5bb1a341b1b90d0 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46010)
I20260926 20:56:19.935972 6515 tablet_copy_service.cc:161] P 896ea7692253410faae3e0185a54e566: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 79c1cf7b052a412db5bb1a341b1b90d0 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46010: session id = 79c1cf7b052a412db5bb1a341b1b90d0-00000000000000000000000000000000
I20260926 20:56:19.936609 6515 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 20:56:19.937086 7147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 20:56:19.938458 7147 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: tablet copy: Starting download of 0 data blocks...
I20260926 20:56:19.938623 7147 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: tablet copy: Starting download of 1 WAL segments...
I20260926 20:56:19.939438 7147 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 20:56:19.940304 6515 tablet_copy_service.cc:342] P 896ea7692253410faae3e0185a54e566: Request end of tablet copy session 79c1cf7b052a412db5bb1a341b1b90d0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46010
I20260926 20:56:19.940362 6515 tablet_copy_service.cc:433] P 896ea7692253410faae3e0185a54e566: ending tablet copy session 79c1cf7b052a412db5bb1a341b1b90d0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 79c1cf7b052a412db5bb1a341b1b90d0
I20260926 20:56:19.944991 7147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:19.946337 7263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:19.946385 7260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:19.946537 7147 server_base.cc:1034] running on GCE node
W20260926 20:56:19.948589 7261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:19.948814 7147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:19.948974 7147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:19.950084 7147 hybrid_clock.cc:648] HybridClock initialized: now 1790456179950067 us; error 38 us; skew 500 ppm
Sep 26 20:56:19 dist-test-slave-fl7g krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456179, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:19.951728 7147 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 20:56:19.952009 7147 webserver.cc:460] Webserver started at http://127.0.98.60:39229/ using document root <none> and password file <none>
I20260926 20:56:19.952136 7147 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:19.952175 7147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:19.953007 7147 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:56:19.953605 7270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:19.953780 7147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:56:19.953838 7147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/wal
uuid: "79c1cf7b052a412db5bb1a341b1b90d0"
format_stamp: "Formatted at 2026-09-26 20:56:19 on dist-test-slave-fl7g"
server_key: "b3c7759bf9dd3cdd8211c5f156f55e93"
server_key_iv: "b3a8f9db909c6cfc35bf92b3ce1579ca"
server_key_version: "encryptionkey@0"
I20260926 20:56:19.953905 7147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:19.973593 7147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:19.975519 7147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:19.978888 7147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35517
I20260926 20:56:19.979004 7322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35517 every 8 connection(s)
I20260926 20:56:19.979225 7147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/master-2/data/info.pb
I20260926 20:56:19.980118 7323 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:56:19.980746 7323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: Bootstrap starting.
I20260926 20:56:19.983340 7323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:56:19.983577 7323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: Bootstrap complete.
I20260926 20:56:19.983796 7323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } }
I20260926 20:56:19.983886 7323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 79c1cf7b052a412db5bb1a341b1b90d0, State: Initialized, Role: LEARNER
I20260926 20:56:19.983923 7323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } }
I20260926 20:56:19.984089 7323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:19.984092 7326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } } }
I20260926 20:56:19.984311 7326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:56:19.985738 7337 catalog_manager.cc:1259] Loaded cluster ID: 8e199d17d12345d1868af6806326bb00
I20260926 20:56:19.985792 7337 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: loading cluster ID for follower catalog manager: success
I20260926 20:56:19.987357 7337 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: acquiring CA information for follower catalog manager: success
I20260926 20:56:19.987692 7337 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 20:56:19.990198 7240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:19.990435 7240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:19.990487 7240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:19.991720 7240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:19.991786 7240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:19.991830 7240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:56:19.993441 7240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42191
--tserver_master_addrs=127.0.98.62:44429,127.0.98.61:40951,127.0.98.60:35517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:19.993710 7240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:19.993944 7240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:56:20.000593 7240 server_base.cc:1034] running on GCE node
W20260926 20:56:20.000564 7342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:20.000564 7343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:20.000872 7345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:20.001132 7240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:20.001406 7240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:20.002579 7240 hybrid_clock.cc:648] HybridClock initialized: now 1790456180002563 us; error 31 us; skew 500 ppm
Sep 26 20:56:20 dist-test-slave-fl7g krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456180, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:20.005725 7240 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 20:56:20.006273 7240 webserver.cc:460] Webserver started at http://127.0.98.1:42191/ using document root <none> and password file <none>
I20260926 20:56:20.006546 7240 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:20.006628 7240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:20.008383 7240 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:56:20.009236 7352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:20.009469 7240 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:20.009577 7240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "a812960f384e4f578bbf8392b67794c5"
format_stamp: "Formatted at 2026-09-26 20:56:18 on dist-test-slave-fl7g"
server_key: "fe8a6fdf23f1eab8b7049ec513b979f1"
server_key_iv: "9dbb788cbda656a3504ffbb04c05292e"
server_key_version: "encryptionkey@0"
I20260926 20:56:20.010018 7240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:20.040393 7240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:20.042855 7240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:20.043033 7240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:20.043289 7240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:20.043608 7240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:20.043658 7240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:20.043700 7240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:20.043731 7240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:20.049998 7240 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42067
I20260926 20:56:20.050079 7465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42067 every 8 connection(s)
I20260926 20:56:20.050436 7240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:56:20.051059 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 7240
I20260926 20:56:20.051229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 6691
I20260926 20:56:20.057947 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46125
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44429
--webserver_port=33245
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44429,127.0.98.61:40951
--tserver_master_addrs=127.0.98.62:44429,127.0.98.61:40951,127.0.98.60:35517 with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:56:20 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456180, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 20:56:20 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456180, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 20:56:20 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456180, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 20:56:20.064085 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.051529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36151 (local address 127.0.98.62:44429)
0926 20:56:20.051580 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.051582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.053631 (+ 2049us) server_negotiation.cc:407] Connection header received
0926 20:56:20.053728 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.053729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.053742 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.053771 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:20.054222 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.055079 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.056815 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.057114 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.057157 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.060588 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:20.060589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:20.060590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:20.060599 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:20.062491 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.063145 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.063146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.063147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.063218 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.063412 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.063413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.063413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.063554 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:20.063614 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.063870 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.063955 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:56:20.064306 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.051221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.1:36151)
0926 20:56:20.053559 (+ 2338us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:20.053571 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:20.053658 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:20.053849 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:20.053850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:20.054056 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:20.054133 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.054135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.055205 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.055206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.056699 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.056702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.057250 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.057251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.057415 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.058582 (+ 1167us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:20.058587 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:20.060508 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:20.062574 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:20.062575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:20.062575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:20.063053 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:20.063287 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:20.063288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:20.063289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:20.063345 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:20.063640 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:20.063641 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:20.063701 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 20:56:20.063766 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:56:20.063998 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2307,"mutex_wait_us":75,"thread_start_us":103,"threads_started":1}
I20260926 20:56:20.064849 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.053680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35517 (local address 127.0.98.1:37285)
0926 20:56:20.053868 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:20.053880 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:20.053930 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:20.055310 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:20.055311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:20.055477 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:20.056890 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.056893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.058080 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.058081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.058885 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.058887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.059433 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.059434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.059511 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.060212 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:20.060217 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:20.061788 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:20.063739 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:20.063740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:20.063740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:20.063972 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:20.064179 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:20.064179 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:20.064180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:20.064237 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:20.064537 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:20.064538 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:20.064581 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 20:56:20.064642 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 20:56:20.064740 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":107,"thread_start_us":72,"threads_started":1}
I20260926 20:56:20.064851 7467 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44429
I20260926 20:56:20.064971 7467 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:20.065163 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.053844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37285 (local address 127.0.98.60:35517)
0926 20:56:20.054212 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.054214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.054220 (+ 6us) server_negotiation.cc:407] Connection header received
0926 20:56:20.054243 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.054243 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.054255 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.054807 (+ 552us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:20.056988 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.057971 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.058984 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.059313 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.059379 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.061888 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:20.061889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:20.061890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:20.061898 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:20.063659 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.064049 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.064050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.064050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.064111 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.064287 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.064288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.064289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.064448 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:20.064513 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.064986 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.065076 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":54,"threads_started":1}
I20260926 20:56:20.065171 7467 heartbeater.cc:511] Master 127.0.98.62:44429 requested a full tablet report, sending...
I20260926 20:56:20.065192 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.051205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39781 (local address 127.0.98.61:40951)
0926 20:56:20.051826 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.051828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.051837 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:56:20.052613 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.052614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.052625 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.052816 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:20.054157 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.055207 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.056938 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.057224 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.057264 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.060223 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:20.060224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:20.060224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:20.060234 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:20.062233 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.062806 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.062808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.062808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.062871 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.063199 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.063200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.063201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.063370 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:20.063428 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.064866 (+ 1438us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.065045 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591}
I20260926 20:56:20.065500 7466 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35517
I20260926 20:56:20.065548 7466 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:20.065626 7466 heartbeater.cc:511] Master 127.0.98.60:35517 requested a full tablet report, sending...
I20260926 20:56:20.065850 6495 ts_manager.cc:194] Re-registered known tserver with Master: a812960f384e4f578bbf8392b67794c5 (127.0.98.1:42067)
I20260926 20:56:20.066008 7279 ts_manager.cc:194] Registered new tserver with Master: a812960f384e4f578bbf8392b67794c5 (127.0.98.1:42067)
I20260926 20:56:20.066427 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36151
I20260926 20:56:20.066779 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.051220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40951 (local address 127.0.98.1:39781)
0926 20:56:20.051656 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:20.051702 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:20.052497 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:20.052919 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:20.052923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:20.053620 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:20.054073 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.054077 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.055283 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.055284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.056699 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.056702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.057336 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.057337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.057415 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.058119 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:20.058130 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:20.060083 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:20.062339 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:20.062348 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:20.062351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:20.062724 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:20.062947 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:20.062948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:20.062948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:20.063110 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:20.063454 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:20.063461 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:20.063641 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 20:56:20.063763 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 20:56:20.066626 (+ 2863us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":73,"thread_start_us":83,"threads_started":1}
I20260926 20:56:20.067344 7468 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40951
I20260926 20:56:20.067404 7468 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:20.067474 7468 heartbeater.cc:511] Master 127.0.98.61:40951 requested a full tablet report, sending...
I20260926 20:56:20.067866 7093 ts_manager.cc:194] Registered new tserver with Master: a812960f384e4f578bbf8392b67794c5 (127.0.98.1:42067)
W20260926 20:56:20.145294 7474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:20.145529 7474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:20.145555 7474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:20.146934 7474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:20.146996 7474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:20.147029 7474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:56:20.148824 7474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33245
--tserver_master_addrs=127.0.98.62:44429,127.0.98.61:40951,127.0.98.60:35517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:20.149055 7474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:20.149243 7474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:20.154588 7480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:20.154584 7479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:20.154584 7482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:20.154650 7474 server_base.cc:1034] running on GCE node
I20260926 20:56:20.155133 7474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:20.155392 7474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:20.156566 7474 hybrid_clock.cc:648] HybridClock initialized: now 1790456180156551 us; error 27 us; skew 500 ppm
Sep 26 20:56:20 dist-test-slave-fl7g krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456180, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:20.159045 7474 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 20:56:20.159552 7474 webserver.cc:460] Webserver started at http://127.0.98.2:33245/ using document root <none> and password file <none>
I20260926 20:56:20.159786 7474 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:20.159864 7474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:20.161324 7474 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:20.161880 7489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:20.162041 7474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:56:20.162135 7474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "2cdb882ced5c4fc383c0ade935a01cab"
format_stamp: "Formatted at 2026-09-26 20:56:19 on dist-test-slave-fl7g"
server_key: "5bf80a061563aacd01d41c6f20d8acfb"
server_key_iv: "801aeec36936670675a5f193cdea943a"
server_key_version: "encryptionkey@0"
I20260926 20:56:20.162489 7474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:20.170212 7474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:20.172537 7474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:20.172700 7474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:20.172940 7474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:20.173273 7474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:20.173316 7474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:20.173350 7474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:20.173379 7474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:20.179745 7474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46125
I20260926 20:56:20.179785 7602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46125 every 8 connection(s)
I20260926 20:56:20.180145 7474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:56:20.186308 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 7474
I20260926 20:56:20.186535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 6825
Sep 26 20:56:20 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456180, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 20:56:20 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456180, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 20:56:20.192255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41389
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44429
--webserver_port=39359
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44429,127.0.98.61:40951
--tserver_master_addrs=127.0.98.62:44429,127.0.98.61:40951,127.0.98.60:35517 with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:56:20.192283 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.181169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36343 (local address 127.0.98.60:35517)
0926 20:56:20.181217 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.181219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.181400 (+ 181us) server_negotiation.cc:407] Connection header received
0926 20:56:20.181885 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.181886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.181901 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.181933 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:20.182707 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.183454 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.184694 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.184936 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.184999 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.189075 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:20.189077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:20.189078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:20.189086 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:20.190754 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.191296 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.191298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.191298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.191356 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.191583 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.191584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.191585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.191733 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:20.191776 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.192079 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.192183 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:56:20.192602 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.180889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35517 (local address 127.0.98.2:36343)
0926 20:56:20.181300 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:20.181311 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:20.181785 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:20.182414 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:20.182415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:20.182553 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:20.182604 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.182605 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.183526 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.183527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.184580 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.184582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.185059 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.185060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.185249 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.186004 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:20.186014 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:20.188863 (+ 2849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:20.190841 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:20.190850 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:20.190852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:20.191207 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:20.191425 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:20.191425 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:20.191425 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:20.191513 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:20.191819 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:20.191820 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:20.191935 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 20:56:20.192033 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 20:56:20.192323 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":153,"thread_start_us":92,"threads_started":1}
I20260926 20:56:20.192634 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.180875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39363 (local address 127.0.98.61:40951)
0926 20:56:20.180920 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.180922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.181249 (+ 327us) server_negotiation.cc:407] Connection header received
0926 20:56:20.181868 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.181870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.181883 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.181938 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:20.182688 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.183453 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.184925 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.185206 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.185269 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.189313 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:20.189315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:20.189315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:20.189325 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:20.190785 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.191259 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.191260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.191261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.191321 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.191579 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.191581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.191581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.191706 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:20.191773 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.192438 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.192544 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:56:20.193147 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.180889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40951 (local address 127.0.98.2:39363)
0926 20:56:20.181080 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:20.181117 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:20.181766 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:20.182019 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:20.182023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:20.182344 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:20.182585 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.182589 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.183970 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.183970 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.184855 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.184857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.185276 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.185277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.185340 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.186197 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:20.186202 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:20.189212 (+ 3010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:20.190884 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:20.190885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:20.190885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:20.191179 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:20.191389 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:20.191390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:20.191390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:20.191503 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:20.191775 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:20.191782 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:20.191930 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 20:56:20.192055 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 20:56:20.192248 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":24,"thread_start_us":89,"threads_started":1}
I20260926 20:56:20.193272 7603 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35517
I20260926 20:56:20.193360 7603 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:20.193585 7603 heartbeater.cc:511] Master 127.0.98.60:35517 requested a full tablet report, sending...
I20260926 20:56:20.193770 7605 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40951
I20260926 20:56:20.193813 7605 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:20.193878 7605 heartbeater.cc:511] Master 127.0.98.61:40951 requested a full tablet report, sending...
I20260926 20:56:20.194039 7279 ts_manager.cc:194] Registered new tserver with Master: 2cdb882ced5c4fc383c0ade935a01cab (127.0.98.2:46125)
I20260926 20:56:20.194150 7093 ts_manager.cc:194] Registered new tserver with Master: 2cdb882ced5c4fc383c0ade935a01cab (127.0.98.2:46125)
Sep 26 20:56:20 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456180, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:56:20.197808 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.181298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51147 (local address 127.0.98.62:44429)
0926 20:56:20.181340 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.181342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.181414 (+ 72us) server_negotiation.cc:407] Connection header received
0926 20:56:20.181885 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.181886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.181899 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.181933 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:20.182709 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.183202 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.184704 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.185003 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.185035 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.194978 (+ 9943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:20.194980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:20.194980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:20.194989 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:20.196655 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.197030 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.197031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.197031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.197095 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.197287 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.197288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.197288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.197412 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:20.197493 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.197633 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.197716 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:56:20.197827 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.181195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.2:51147)
0926 20:56:20.181337 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:20.181345 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:20.181785 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:20.182021 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:20.182024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:20.182358 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:20.182586 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.182593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.183296 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.183297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.184583 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.184585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.185115 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.185116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.185253 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.186055 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:20.186060 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:20.194906 (+ 8846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:20.196756 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:20.196757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:20.196757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:20.196927 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:20.197157 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:20.197158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:20.197158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:20.197219 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:20.197477 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:20.197478 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:20.197523 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 20:56:20.197606 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 20:56:20.197717 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":3,"thread_start_us":57,"threads_started":1}
I20260926 20:56:20.198269 7604 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44429
I20260926 20:56:20.198319 7604 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:20.198383 7604 heartbeater.cc:511] Master 127.0.98.62:44429 requested a full tablet report, sending...
I20260926 20:56:20.198675 6495 ts_manager.cc:194] Re-registered known tserver with Master: 2cdb882ced5c4fc383c0ade935a01cab (127.0.98.2:46125)
I20260926 20:56:20.199239 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51147
W20260926 20:56:20.277158 7610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:20.277376 7610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:20.277401 7610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:20.278520 7610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:20.278559 7610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:20.278586 7610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:56:20.280081 7610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39359
--tserver_master_addrs=127.0.98.62:44429,127.0.98.61:40951,127.0.98.60:35517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:20.280282 7610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:20.280591 7610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:20.285739 7615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:20.285739 7618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:20.285733 7616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:20.286087 7610 server_base.cc:1034] running on GCE node
I20260926 20:56:20.286314 7610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:20.286581 7610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:20.287736 7610 hybrid_clock.cc:648] HybridClock initialized: now 1790456180287726 us; error 33 us; skew 500 ppm
Sep 26 20:56:20 dist-test-slave-fl7g krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456180, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:20.290417 7610 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 20:56:20.290858 7610 webserver.cc:460] Webserver started at http://127.0.98.3:39359/ using document root <none> and password file <none>
I20260926 20:56:20.291108 7610 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:20.291167 7610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:20.292675 7610 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:20.293295 7625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:20.293502 7610 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:20.293597 7610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "7b23b75bbf57466498a372ea17df29bf"
format_stamp: "Formatted at 2026-09-26 20:56:19 on dist-test-slave-fl7g"
server_key: "d71b70502be78506296e5723a1c6c6ff"
server_key_iv: "0913f004b85177bed916d1a0a15de83b"
server_key_version: "encryptionkey@0"
I20260926 20:56:20.293943 7610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:20.310097 7610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:20.312664 7610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:20.312813 7610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:20.313071 7610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:20.313397 7610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:20.313432 7610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:20.313457 7610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:20.313469 7610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:20.319725 7610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41389
I20260926 20:56:20.319769 7738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41389 every 8 connection(s)
I20260926 20:56:20.320153 7610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:56:20 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456180, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 20:56:20 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456180, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:56:20.328079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 7610
Sep 26 20:56:20 dist-test-slave-fl7g krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456180, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 20:56:20.331622 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.321463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42055 (local address 127.0.98.62:44429)
0926 20:56:20.321520 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.321522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.321539 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:56:20.322173 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.322174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.322193 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.322249 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:20.322986 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.323752 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.324908 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.325178 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.325242 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.328007 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:20.328009 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:20.328009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:20.328018 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:20.329910 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.330400 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.330401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.330402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.330468 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.330792 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.330793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.330793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.330955 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:20.331012 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.331427 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.331517 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 20:56:20.331759 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.320932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52295 (local address 127.0.98.60:35517)
0926 20:56:20.320980 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.320983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.321266 (+ 283us) server_negotiation.cc:407] Connection header received
0926 20:56:20.322047 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.322049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.322058 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.322118 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:20.322987 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.323746 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.324892 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.325163 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.325231 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.327543 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:20.327545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:20.327545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:20.327553 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:20.330512 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.330932 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.330934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.330934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.330995 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.331196 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.331197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.331197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.331324 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:20.331369 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.331594 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.331677 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:56:20.331879 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.320960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40951 (local address 127.0.98.3:52185)
0926 20:56:20.321149 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:20.321185 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:20.321820 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:20.322411 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:20.322412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:20.322651 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:20.322871 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.322876 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.323582 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.323583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.324636 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.324638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.325074 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.325075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.325220 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.325727 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:20.325735 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:20.328527 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:20.330423 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:20.330424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:20.330425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:20.330671 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:20.330876 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:20.330876 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:20.330877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:20.330933 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:20.331234 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:20.331235 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:20.331282 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 20:56:20.331377 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 20:56:20.331602 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":4,"thread_start_us":117,"threads_started":1}
I20260926 20:56:20.332121 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.320960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35517 (local address 127.0.98.3:52295)
0926 20:56:20.321141 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:20.321185 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:20.321800 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:20.322137 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:20.322142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:20.322661 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:20.322872 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.322878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.323847 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.323847 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.324800 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.324803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.325245 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.325246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.325312 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.326000 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:20.326005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:20.327430 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:20.330647 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:20.330648 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:20.330648 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:20.330850 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:20.331067 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:20.331067 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:20.331067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:20.331123 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:20.331417 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:20.331417 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:20.331465 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 20:56:20.331922 (+ 457us) client_negotiation.cc:240] Negotiation successful
0926 20:56:20.332015 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":60,"thread_start_us":108,"threads_started":1}
I20260926 20:56:20.332144 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.321285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.3:42055)
0926 20:56:20.321419 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:20.321430 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:20.321816 (+ 386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:20.322310 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:20.322310 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:20.322667 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:20.322872 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.322878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.323847 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.323847 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.324805 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.324806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.325297 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.325298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.325360 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.325821 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:20.325826 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:20.327881 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:20.330013 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:20.330022 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:20.330025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:20.330303 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:20.330540 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:20.330541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:20.330541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:20.330701 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:20.331065 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:20.331072 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:20.331260 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 20:56:20.331702 (+ 442us) client_negotiation.cc:240] Negotiation successful
0926 20:56:20.331807 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":54,"thread_start_us":65,"threads_started":1}
I20260926 20:56:20.332538 7739 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35517
I20260926 20:56:20.332660 7739 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:20.332712 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.320939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52185 (local address 127.0.98.61:40951)
0926 20:56:20.320988 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.320990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.321272 (+ 282us) server_negotiation.cc:407] Connection header received
0926 20:56:20.321916 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.321918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.321932 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.321987 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:20.322973 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.323475 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.324734 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.324995 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.325058 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.328626 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:20.328628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:20.328628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:20.328636 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:20.330320 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.330750 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.330751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.330751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.330808 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.331002 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.331002 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.331003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.331145 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:20.331204 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.332499 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.332600 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:56:20.332873 7739 heartbeater.cc:511] Master 127.0.98.60:35517 requested a full tablet report, sending...
I20260926 20:56:20.332890 7740 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44429
I20260926 20:56:20.332937 7740 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:20.333010 7740 heartbeater.cc:511] Master 127.0.98.62:44429 requested a full tablet report, sending...
I20260926 20:56:20.333186 7741 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40951
I20260926 20:56:20.333233 7741 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:20.333292 7741 heartbeater.cc:511] Master 127.0.98.61:40951 requested a full tablet report, sending...
I20260926 20:56:20.333309 7279 ts_manager.cc:194] Registered new tserver with Master: 7b23b75bbf57466498a372ea17df29bf (127.0.98.3:41389)
I20260926 20:56:20.333454 6495 ts_manager.cc:194] Re-registered known tserver with Master: 7b23b75bbf57466498a372ea17df29bf (127.0.98.3:41389)
I20260926 20:56:20.333556 7093 ts_manager.cc:194] Registered new tserver with Master: 7b23b75bbf57466498a372ea17df29bf (127.0.98.3:41389)
I20260926 20:56:20.333922 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42055
I20260926 20:56:20.339439 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.330010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.0.98.61:40951)
0926 20:56:20.330114 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.330116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.331360 (+ 1244us) server_negotiation.cc:407] Connection header received
0926 20:56:20.331573 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.331574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.331588 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.331627 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:20.333175 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.333906 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.334726 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.334912 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.334998 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.336505 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:20.336506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:20.336507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:20.336514 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:20.338224 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.338654 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.338655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.338655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.338721 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.338925 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.338926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.338926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.339041 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:20.339146 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.339241 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.339327 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":53,"threads_started":1}
I20260926 20:56:20.342130 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.331191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.0.98.60:35517)
0926 20:56:20.331833 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.331835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.334982 (+ 3147us) server_negotiation.cc:407] Connection header received
0926 20:56:20.335095 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.335096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.335109 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.335171 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:20.335727 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.336561 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.337580 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.337879 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.337949 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.339478 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:20.339480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:20.339480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:20.339488 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:20.341132 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.341442 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.341443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.341444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.341495 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.341640 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.341641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.341642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.341754 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:20.341826 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.341946 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.342025 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":51,"threads_started":1}
I20260926 20:56:20.342693 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.330667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46024 (local address 127.0.98.62:44429)
0926 20:56:20.330802 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.330804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.334846 (+ 4042us) server_negotiation.cc:407] Connection header received
0926 20:56:20.334911 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.334913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.334926 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.334992 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:20.335736 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.336227 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.337238 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.337767 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.337884 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.339557 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:20.339558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:20.339558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:20.339566 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:20.341301 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.341851 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.341852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.341852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.341900 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.342114 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.342115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.342115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.342246 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:20.342335 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.342541 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.342622 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260926 20:56:20.344288 6495 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:56:20.346029 6495 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:56:20.362020 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.358287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46125 (local address 127.0.0.1:35994)
0926 20:56:20.358712 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:20.358723 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:20.358780 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:20.358995 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:20.358996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:20.358999 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:20.359054 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.359056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.360111 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.360112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.361051 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.361054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.361739 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.361740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.361807 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.361812 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:56:20.361873 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 20:56:20.361918 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":143,"thread_start_us":100,"threads_started":1}
I20260926 20:56:20.362392 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.358546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.0.98.2:46125)
0926 20:56:20.358732 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.358737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.358828 (+ 91us) server_negotiation.cc:407] Connection header received
0926 20:56:20.358872 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.358880 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.358895 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.358992 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:20.359137 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.360027 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.361135 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.361647 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.361741 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.361817 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.361889 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.361931 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":74,"threads_started":1}
I20260926 20:56:20.363373 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.358278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41389 (local address 127.0.0.1:57448)
0926 20:56:20.358318 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:20.358332 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:20.358375 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:20.359007 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:20.359008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:20.359010 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:20.359072 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.359074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.360852 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.360853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.362001 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.362003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.363120 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.363121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.363186 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.363190 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:56:20.363245 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 20:56:20.363296 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 20:56:20.363373 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.358503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.0.98.1:42067)
0926 20:56:20.358894 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.358898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.358908 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:56:20.358948 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.358950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.358967 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.359093 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:20.359208 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.360492 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.361986 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.362864 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.362945 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.363040 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.363246 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.363287 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":103,"threads_started":1}
I20260926 20:56:20.363379 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.358398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42067 (local address 127.0.0.1:54914)
0926 20:56:20.358610 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:20.358622 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:20.358672 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:20.359089 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:20.359090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:20.359092 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:20.359138 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.359140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.360594 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.360595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.361879 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.361881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.363079 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.363080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.363164 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.363169 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:56:20.363229 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 20:56:20.363276 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":72,"threads_started":1}
I20260926 20:56:20.363263 7537 tablet_service.cc:1467] Processing CreateTablet for tablet 19d6ebeeae644919b84e826ffb1c49e7 (DEFAULT_TABLE table=test-table [id=f9c4663199724052a3faae29012b945b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:20.363691 7537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d6ebeeae644919b84e826ffb1c49e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:20.363926 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.358367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.0.98.3:41389)
0926 20:56:20.358808 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.358814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.358826 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:56:20.358872 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.358875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.358893 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.359152 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:20.359213 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.360759 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.362226 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.362988 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.363099 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.363158 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.363801 (+ 643us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.363851 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":106,"threads_started":1}
I20260926 20:56:20.364177 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 19d6ebeeae644919b84e826ffb1c49e7 (DEFAULT_TABLE table=test-table [id=f9c4663199724052a3faae29012b945b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:20.364534 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d6ebeeae644919b84e826ffb1c49e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:20.365074 7673 tablet_service.cc:1467] Processing CreateTablet for tablet 19d6ebeeae644919b84e826ffb1c49e7 (DEFAULT_TABLE table=test-table [id=f9c4663199724052a3faae29012b945b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:20.365413 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d6ebeeae644919b84e826ffb1c49e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:20.367138 7765 tablet_bootstrap.cc:492] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab: Bootstrap starting.
I20260926 20:56:20.367928 7765 tablet_bootstrap.cc:654] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:20.368260 7765 log.cc:824] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:20.368957 7767 tablet_bootstrap.cc:492] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5: Bootstrap starting.
I20260926 20:56:20.369684 7767 tablet_bootstrap.cc:654] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:20.369951 7768 tablet_bootstrap.cc:492] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf: Bootstrap starting.
I20260926 20:56:20.370057 7767 log.cc:824] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:20.370672 7768 tablet_bootstrap.cc:654] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:20.370710 7765 tablet_bootstrap.cc:492] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab: No bootstrap required, opened a new log
I20260926 20:56:20.370784 7765 ts_tablet_manager.cc:1397] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 20:56:20.370915 7767 tablet_bootstrap.cc:492] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5: No bootstrap required, opened a new log
I20260926 20:56:20.370981 7767 ts_tablet_manager.cc:1397] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:56:20.371130 7768 log.cc:824] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:20.371829 7768 tablet_bootstrap.cc:492] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf: No bootstrap required, opened a new log
I20260926 20:56:20.371924 7768 ts_tablet_manager.cc:1397] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:56:20.373219 7767 raft_consensus.cc:348] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } }
I20260926 20:56:20.373814 7767 raft_consensus.cc:374] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:20.373859 7767 raft_consensus.cc:729] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a812960f384e4f578bbf8392b67794c5, State: Initialized, Role: FOLLOWER
I20260926 20:56:20.373878 7765 raft_consensus.cc:348] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } }
I20260926 20:56:20.374042 7765 raft_consensus.cc:374] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:20.373950 7768 raft_consensus.cc:348] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } }
I20260926 20:56:20.374015 7767 consensus_queue.cc:260] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } }
I20260926 20:56:20.374078 7765 raft_consensus.cc:729] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cdb882ced5c4fc383c0ade935a01cab, State: Initialized, Role: FOLLOWER
I20260926 20:56:20.374097 7768 raft_consensus.cc:374] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:20.374125 7768 raft_consensus.cc:729] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b23b75bbf57466498a372ea17df29bf, State: Initialized, Role: FOLLOWER
I20260926 20:56:20.374228 7768 consensus_queue.cc:260] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } }
I20260926 20:56:20.374215 7765 consensus_queue.cc:260] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } }
I20260926 20:56:20.374328 7767 ts_tablet_manager.cc:1428] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 20:56:20.374473 7765 ts_tablet_manager.cc:1428] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 20:56:20.374480 7768 ts_tablet_manager.cc:1428] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:56:20.374797 7740 heartbeater.cc:503] Master 127.0.98.62:44429 was elected leader, sending a full tablet report...
I20260926 20:56:20.375247 7467 heartbeater.cc:503] Master 127.0.98.62:44429 was elected leader, sending a full tablet report...
I20260926 20:56:20.376312 7604 heartbeater.cc:503] Master 127.0.98.62:44429 was elected leader, sending a full tablet report...
I20260926 20:56:20.376889 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.372814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35517 (local address 127.0.0.1:37544)
0926 20:56:20.372922 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:20.372933 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:20.372995 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:20.373176 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:20.373177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:20.373179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:20.373233 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.373235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.374605 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.374606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.375663 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.375667 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.376625 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.376626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.376689 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.376692 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:56:20.376744 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 20:56:20.376795 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 20:56:20.377171 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.372931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.0.98.60:35517)
0926 20:56:20.372989 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.372991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.373018 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:56:20.373052 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.373053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.373079 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.373165 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:20.373661 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.374403 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.375786 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.376401 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.376489 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.376565 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.377058 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.377105 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 20:56:20.377444 7294 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260926 20:56:20.377858 7326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 896ea7692253410faae3e0185a54e566. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } } }
I20260926 20:56:20.377950 7326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: This master's current role is: LEARNER
W20260926 20:56:20.430888 7606 tablet.cc:2369] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:56:20.556041 7469 tablet.cc:2369] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:56:20.571187 7742 tablet.cc:2369] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:56:20.590245 7773 raft_consensus.cc:482] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:56:20.590376 7773 raft_consensus.cc:504] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } }
I20260926 20:56:20.590816 7773 leader_election.cc:288] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b23b75bbf57466498a372ea17df29bf (127.0.98.3:41389), a812960f384e4f578bbf8392b67794c5 (127.0.98.1:42067)
I20260926 20:56:20.593753 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.591150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60891 (local address 127.0.98.3:41389)
0926 20:56:20.591212 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.591214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.591231 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:56:20.591294 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.591295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.591308 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.591384 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:20.591553 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.592026 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.593039 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.593395 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.593481 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.593492 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.593646 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.593683 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:56:20.593763 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.591075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41389 (local address 127.0.98.2:60891)
0926 20:56:20.591146 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:20.591159 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:20.591206 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:20.591388 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:20.591389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:20.591391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:20.591471 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.591474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.592112 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.592113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.592929 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.592931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.593470 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.593471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.593561 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.593564 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:56:20.593636 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 20:56:20.593678 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 20:56:20.593771 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.591280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34267 (local address 127.0.98.1:42067)
0926 20:56:20.591368 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.591370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.591379 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:56:20.591410 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.591411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.591421 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.591529 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:20.591673 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.592134 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.593062 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.593414 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.593509 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.593519 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.593669 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.593705 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:56:20.593786 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.591237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42067 (local address 127.0.98.2:34267)
0926 20:56:20.591289 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:20.591298 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:20.591328 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:20.591536 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:20.591536 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:20.591538 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:20.591602 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.591604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.592214 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.592215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.592972 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:20.592974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.593511 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.593512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:20.593592 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.593594 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:56:20.593651 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 20:56:20.593690 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 20:56:20.594185 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d6ebeeae644919b84e826ffb1c49e7" candidate_uuid: "2cdb882ced5c4fc383c0ade935a01cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a812960f384e4f578bbf8392b67794c5" is_pre_election: true
I20260926 20:56:20.594184 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d6ebeeae644919b84e826ffb1c49e7" candidate_uuid: "2cdb882ced5c4fc383c0ade935a01cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b23b75bbf57466498a372ea17df29bf" is_pre_election: true
I20260926 20:56:20.594352 7420 raft_consensus.cc:2454] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cdb882ced5c4fc383c0ade935a01cab in term 0.
I20260926 20:56:20.594352 7693 raft_consensus.cc:2454] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cdb882ced5c4fc383c0ade935a01cab in term 0.
I20260926 20:56:20.594615 7490 leader_election.cc:302] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cdb882ced5c4fc383c0ade935a01cab, 7b23b75bbf57466498a372ea17df29bf; no voters:
I20260926 20:56:20.594761 7773 raft_consensus.cc:2781] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:56:20.594823 7773 raft_consensus.cc:482] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:56:20.594841 7773 raft_consensus.cc:3037] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:20.595597 7773 raft_consensus.cc:504] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } }
I20260926 20:56:20.595710 7773 leader_election.cc:288] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [CANDIDATE]: Term 1 election: Requested vote from peers 7b23b75bbf57466498a372ea17df29bf (127.0.98.3:41389), a812960f384e4f578bbf8392b67794c5 (127.0.98.1:42067)
I20260926 20:56:20.595889 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d6ebeeae644919b84e826ffb1c49e7" candidate_uuid: "2cdb882ced5c4fc383c0ade935a01cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b23b75bbf57466498a372ea17df29bf"
I20260926 20:56:20.595932 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d6ebeeae644919b84e826ffb1c49e7" candidate_uuid: "2cdb882ced5c4fc383c0ade935a01cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a812960f384e4f578bbf8392b67794c5"
I20260926 20:56:20.595973 7693 raft_consensus.cc:3037] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:20.596036 7420 raft_consensus.cc:3037] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:20.596825 7693 raft_consensus.cc:2454] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cdb882ced5c4fc383c0ade935a01cab in term 1.
I20260926 20:56:20.596966 7420 raft_consensus.cc:2454] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cdb882ced5c4fc383c0ade935a01cab in term 1.
I20260926 20:56:20.596978 7490 leader_election.cc:302] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cdb882ced5c4fc383c0ade935a01cab, 7b23b75bbf57466498a372ea17df29bf; no voters:
I20260926 20:56:20.597069 7773 raft_consensus.cc:2781] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:20.597229 7773 raft_consensus.cc:686] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [term 1 LEADER]: Becoming Leader. State: Replica: 2cdb882ced5c4fc383c0ade935a01cab, State: Running, Role: LEADER
I20260926 20:56:20.597335 7773 consensus_queue.cc:237] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } }
I20260926 20:56:20.598169 6494 catalog_manager.cc:5495] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cdb882ced5c4fc383c0ade935a01cab (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2cdb882ced5c4fc383c0ade935a01cab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } health_report { overall_health: UNKNOWN } } }
I20260926 20:56:20.598820 7248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: true } } }
I20260926 20:56:20.598927 7248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:20.599795 7248 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566: attempting to promote NON_VOTER 79c1cf7b052a412db5bb1a341b1b90d0 to VOTER
I20260926 20:56:20.599922 7248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: false } }
I20260926 20:56:20.600349 7103 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 1 FOLLOWER]: Refusing update from remote peer 896ea7692253410faae3e0185a54e566: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 20:56:20.600399 7294 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 1 LEARNER]: Refusing update from remote peer 896ea7692253410faae3e0185a54e566: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 20:56:20.600570 7781 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260926 20:56:20.600654 7779 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260926 20:56:20.601219 7248 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 79c1cf7b052a412db5bb1a341b1b90d0 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: false } } }
I20260926 20:56:20.601368 7103 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 79c1cf7b052a412db5bb1a341b1b90d0 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: false } } }
I20260926 20:56:20.602093 7248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: false } } }
I20260926 20:56:20.602180 7248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:20.602169 7782 sys_catalog.cc:455] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: false } } }
I20260926 20:56:20.602258 7782 sys_catalog.cc:458] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:56:20.602298 7248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: false } } }
I20260926 20:56:20.602340 7781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: false } } }
I20260926 20:56:20.602384 7248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:20.602399 7781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:20.602542 7294 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 79c1cf7b052a412db5bb1a341b1b90d0 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: false } } }
I20260926 20:56:20.603340 7326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "896ea7692253410faae3e0185a54e566" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: false } } }
I20260926 20:56:20.603447 7326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:56:20.645186 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.642899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36008 (local address 127.0.98.2:46125)
0926 20:56:20.642944 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.642946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.642989 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:56:20.643024 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.643025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.643045 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.643102 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:20.643222 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.643714 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.644449 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.644656 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.644713 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.644881 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:20.644995 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:20.645082 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.645097 (+ 15us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.645126 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:56:20.647117 7693 raft_consensus.cc:1261] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf [term 1 FOLLOWER]: Refusing update from remote peer 2cdb882ced5c4fc383c0ade935a01cab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:20.647131 7420 raft_consensus.cc:1261] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5 [term 1 FOLLOWER]: Refusing update from remote peer 2cdb882ced5c4fc383c0ade935a01cab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:20.647483 7777 consensus_queue.cc:1035] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:20.647691 7773 consensus_queue.cc:1035] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:20.650650 7786 mvcc.cc:205] Tried to move back new op lower bound from 7333708515927896064 to 7333708515727278080. Current Snapshot: MvccSnapshot[applied={T|T < 7333708515927896064}]
I20260926 20:56:20.653694 6494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:20.653782 6494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:20.655201 6494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 896ea7692253410faae3e0185a54e566: Sending DeleteTablet for 3 replicas of tablet 19d6ebeeae644919b84e826ffb1c49e7
I20260926 20:56:20.655490 7537 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d6ebeeae644919b84e826ffb1c49e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35994
I20260926 20:56:20.656234 7673 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d6ebeeae644919b84e826ffb1c49e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57448
I20260926 20:56:20.656339 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d6ebeeae644919b84e826ffb1c49e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54914
I20260926 20:56:20.656852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 6463
I20260926 20:56:20.657624 7796 tablet_replica.cc:330] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf: stopping tablet replica
I20260926 20:56:20.657750 7796 raft_consensus.cc:2229] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:20.657994 7794 tablet_replica.cc:330] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab: stopping tablet replica
I20260926 20:56:20.658085 7794 raft_consensus.cc:2229] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:56:20.658181 7794 raft_consensus.cc:2258] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:20.658281 7796 raft_consensus.cc:2258] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:20.658782 7794 ts_tablet_manager.cc:1905] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:20.660513 7797 tablet_replica.cc:330] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5: stopping tablet replica
I20260926 20:56:20.660693 7797 raft_consensus.cc:2229] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:20.660800 7796 ts_tablet_manager.cc:1905] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:20.660920 7797 raft_consensus.cc:2258] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:20.662710 7796 ts_tablet_manager.cc:1918] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:20.662793 7796 log.cc:1196] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/19d6ebeeae644919b84e826ffb1c49e7
I20260926 20:56:20.663060 7796 ts_tablet_manager.cc:1939] T 19d6ebeeae644919b84e826ffb1c49e7 P 7b23b75bbf57466498a372ea17df29bf: Deleting consensus metadata
I20260926 20:56:20.663627 7794 ts_tablet_manager.cc:1918] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:20.663712 7794 log.cc:1196] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/19d6ebeeae644919b84e826ffb1c49e7
I20260926 20:56:20.663968 7794 ts_tablet_manager.cc:1939] T 19d6ebeeae644919b84e826ffb1c49e7 P 2cdb882ced5c4fc383c0ade935a01cab: Deleting consensus metadata
W20260926 20:56:20.665405 7626 connection.cc:535] server connection from 127.0.0.1:57448 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 20:56:20.665422 7491 connection.cc:535] server connection from 127.0.0.1:35994 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:56:20.666579 7797 ts_tablet_manager.cc:1905] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:20.668385 7797 ts_tablet_manager.cc:1918] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:20.668550 7797 log.cc:1196] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-0/wal/wals/19d6ebeeae644919b84e826ffb1c49e7
I20260926 20:56:20.668807 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.656289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.0.98.60:35517)
0926 20:56:20.656332 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.656334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.656343 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:56:20.656366 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.656367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.656379 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.656475 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:20.657539 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.658325 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.659670 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.659969 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.660032 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.665438 (+ 5406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:20.665440 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:20.665440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:20.665449 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:20.667273 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.667711 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.667713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.667713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.667780 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.667993 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.667994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.667994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.668154 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:20.668294 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.668594 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.668683 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:56:20.669484 7797 ts_tablet_manager.cc:1939] T 19d6ebeeae644919b84e826ffb1c49e7 P a812960f384e4f578bbf8392b67794c5: Deleting consensus metadata
W20260926 20:56:20.669906 7355 connection.cc:441] server connection from 127.0.0.1:54914 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54914 (request call id 1) could send its response
I20260926 20:56:20.675714 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.667459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41152 (local address 127.0.98.61:40951)
0926 20:56:20.667502 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.667504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.668263 (+ 759us) server_negotiation.cc:407] Connection header received
0926 20:56:20.668366 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.668367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.668384 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.668442 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:20.669146 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.669957 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.671022 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.671362 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.671442 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.672916 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:20.672917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:20.672918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:20.672925 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:20.674608 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.674925 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.674926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.674926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.674981 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.675174 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.675175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.675176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.675292 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:20.675392 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.675496 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.675586 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:56:20.676700 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:20.668004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37570 (local address 127.0.98.60:35517)
0926 20:56:20.668045 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:20.668047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:20.669334 (+ 1287us) server_negotiation.cc:407] Connection header received
0926 20:56:20.669424 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:20.669425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:20.669436 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:20.669471 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:20.670193 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.670931 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.671730 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:20.672020 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:20.672093 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:20.673538 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:20.673540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:20.673540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:20.673547 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:20.675368 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.675820 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.675821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.675821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.675879 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:20.676061 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:20.676062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:20.676062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:20.676215 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:20.676299 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:20.676480 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 20:56:20.676565 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:56:21.652202 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:21.651863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.3:45887)
0926 20:56:21.651989 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:21.652068 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260926 20:56:21.652652 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:21.652322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.2:51251)
0926 20:56:21.652519 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:21.652596 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":116,"threads_started":1}
W20260926 20:56:21.652640 7740 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44429 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
I20260926 20:56:21.653043 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:21.652711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.1:39677)
0926 20:56:21.652927 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:21.652999 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":115,"threads_started":1}
W20260926 20:56:21.653090 7604 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44429 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
W20260926 20:56:21.653324 7467 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44429 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
I20260926 20:56:22.155754 7808 raft_consensus.cc:482] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 896ea7692253410faae3e0185a54e566)
I20260926 20:56:22.155965 7808 raft_consensus.cc:504] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: false } }
I20260926 20:56:22.156044 7809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 896ea7692253410faae3e0185a54e566)
I20260926 20:56:22.156185 7809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: false } }
I20260926 20:56:22.156306 7808 leader_election.cc:288] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 896ea7692253410faae3e0185a54e566 (127.0.98.62:44429), 79c1cf7b052a412db5bb1a341b1b90d0 (127.0.98.60:35517)
I20260926 20:56:22.156464 7809 leader_election.cc:288] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 896ea7692253410faae3e0185a54e566 (127.0.98.62:44429), b30454144cd14defb1a3d00fb2cb347e (127.0.98.61:40951)
I20260926 20:56:22.156774 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.156539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.0.1:46520)
0926 20:56:22.156672 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.156725 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
W20260926 20:56:22.156894 7272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
I20260926 20:56:22.156950 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.156545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.0.1:46514)
0926 20:56:22.156847 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.156902 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":47,"threads_started":1}
W20260926 20:56:22.157066 7078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
I20260926 20:56:22.157351 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.157202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.0.1:46528)
0926 20:56:22.157296 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.157323 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":55,"threads_started":1}
W20260926 20:56:22.157418 7272 leader_election.cc:334] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 896ea7692253410faae3e0185a54e566 (127.0.98.62:44429): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
I20260926 20:56:22.157768 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.157660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.0.1:46540)
0926 20:56:22.157715 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.157744 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 20:56:22.157862 7078 leader_election.cc:334] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 896ea7692253410faae3e0185a54e566 (127.0.98.62:44429): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
I20260926 20:56:22.159794 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.156625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40951 (local address 127.0.0.1:41168)
0926 20:56:22.156897 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.156908 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:22.157074 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:22.157262 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:22.157263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:22.157265 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:22.157320 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:22.157323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.158175 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.158176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:22.158965 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:22.158967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.159546 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.159547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:22.159609 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:22.159613 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:56:22.159684 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:56:22.159730 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":47,"threads_started":1}
I20260926 20:56:22.159792 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.156700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41168 (local address 127.0.98.61:40951)
0926 20:56:22.156797 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:22.156799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:22.157036 (+ 237us) server_negotiation.cc:407] Connection header received
0926 20:56:22.157168 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:22.157169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:22.157180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:22.157259 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:22.157390 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.158100 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.159052 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.159461 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.159543 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:22.159555 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:22.159691 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 20:56:22.159731 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":48,"threads_started":1}
I20260926 20:56:22.159909 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.156692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35517 (local address 127.0.0.1:37578)
0926 20:56:22.156909 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.156919 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:22.156975 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:22.157452 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:22.157453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:22.157454 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:22.157495 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:22.157497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.158305 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.158306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:22.159046 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:22.159048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.159646 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.159647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:22.159710 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:22.159714 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:56:22.159781 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 20:56:22.159824 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":69,"threads_started":1}
I20260926 20:56:22.160000 7103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b30454144cd14defb1a3d00fb2cb347e" is_pre_election: true
I20260926 20:56:22.160090 7103 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79c1cf7b052a412db5bb1a341b1b90d0 in term 1.
I20260926 20:56:22.160236 7274 leader_election.cc:302] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79c1cf7b052a412db5bb1a341b1b90d0, b30454144cd14defb1a3d00fb2cb347e; no voters: 896ea7692253410faae3e0185a54e566
I20260926 20:56:22.160341 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.157054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37578 (local address 127.0.98.60:35517)
0926 20:56:22.157326 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:22.157327 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:22.157334 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:56:22.157351 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:22.157352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:22.157361 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:22.157560 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:22.157591 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.158227 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.159135 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.159543 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.159642 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:22.159653 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:22.159785 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 20:56:22.159824 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":74,"threads_started":1}
I20260926 20:56:22.160348 7809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 20:56:22.160427 7809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 896ea7692253410faae3e0185a54e566)
I20260926 20:56:22.160457 7809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:56:22.160539 7294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b30454144cd14defb1a3d00fb2cb347e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" is_pre_election: true
I20260926 20:56:22.161310 7809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: false } }
I20260926 20:56:22.161437 7294 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b30454144cd14defb1a3d00fb2cb347e in current term 2: Already voted for candidate 79c1cf7b052a412db5bb1a341b1b90d0 in this term.
I20260926 20:56:22.161456 7809 leader_election.cc:288] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [CANDIDATE]: Term 2 election: Requested vote from peers 896ea7692253410faae3e0185a54e566 (127.0.98.62:44429), b30454144cd14defb1a3d00fb2cb347e (127.0.98.61:40951)
I20260926 20:56:22.161613 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.161524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.0.1:46558)
0926 20:56:22.161558 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.161589 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 20:56:22.161648 7078 leader_election.cc:302] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b30454144cd14defb1a3d00fb2cb347e; no voters: 79c1cf7b052a412db5bb1a341b1b90d0, 896ea7692253410faae3e0185a54e566
I20260926 20:56:22.161631 7103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b30454144cd14defb1a3d00fb2cb347e"
I20260926 20:56:22.161731 7103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:56:22.162345 7103 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79c1cf7b052a412db5bb1a341b1b90d0 in term 2.
I20260926 20:56:22.162424 7808 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:56:22.162524 7274 leader_election.cc:302] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79c1cf7b052a412db5bb1a341b1b90d0, b30454144cd14defb1a3d00fb2cb347e; no voters:
I20260926 20:56:22.162611 7809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:56:22.162673 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.162566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.0.1:46568)
0926 20:56:22.162617 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.162647 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 20:56:22.162750 7272 leader_election.cc:334] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 896ea7692253410faae3e0185a54e566 (127.0.98.62:44429): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
I20260926 20:56:22.162756 7809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [term 2 LEADER]: Becoming Leader. State: Replica: 79c1cf7b052a412db5bb1a341b1b90d0, State: Running, Role: LEADER
I20260926 20:56:22.162846 7809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: false } }
I20260926 20:56:22.163160 7819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79c1cf7b052a412db5bb1a341b1b90d0. Latest consensus state: current_term: 2 leader_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: false } } }
I20260926 20:56:22.163254 7819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:22.163430 7821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:22.163820 7821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:22.163872 7821 catalog_manager.cc:1259] Loaded cluster ID: 8e199d17d12345d1868af6806326bb00
I20260926 20:56:22.163887 7821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:56:22.164013 7821 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:22.164062 7821 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: Loaded TSK: 0
I20260926 20:56:22.164160 7821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:56:22.217733 7279 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:56:22.219416 7279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:56:22.220521 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.220383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.0.1:46586)
0926 20:56:22.220460 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.220495 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 20:56:22.220567 7103 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [term 2 FOLLOWER]: Refusing update from remote peer 79c1cf7b052a412db5bb1a341b1b90d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 20:56:22.220834 7809 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 20:56:22.220916 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.220832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.0.1:46588)
0926 20:56:22.220865 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.220890 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260926 20:56:22.220992 7272 consensus_peers.cc:487] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 -> Peer 896ea7692253410faae3e0185a54e566 (127.0.98.62:44429): Couldn't send request to peer 896ea7692253410faae3e0185a54e566. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 20:56:22.221470 7808 sys_catalog.cc:455] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79c1cf7b052a412db5bb1a341b1b90d0. Latest consensus state: current_term: 2 leader_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: false } } }
I20260926 20:56:22.221566 7808 sys_catalog.cc:458] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:56:22.221982 7819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: false } } }
I20260926 20:56:22.222074 7819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:22.222144 7808 sys_catalog.cc:455] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "896ea7692253410faae3e0185a54e566" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "b30454144cd14defb1a3d00fb2cb347e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40951 } attrs { promote: false } } peers { permanent_uuid: "79c1cf7b052a412db5bb1a341b1b90d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35517 } attrs { promote: false } } }
I20260926 20:56:22.222236 7808 sys_catalog.cc:458] T 00000000000000000000000000000000 P b30454144cd14defb1a3d00fb2cb347e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:56:22.226840 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.223832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41389 (local address 127.0.0.1:57454)
0926 20:56:22.223892 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.223903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:22.223951 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:22.224257 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:22.224258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:22.224259 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:22.224305 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:22.224306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.225290 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.225291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:22.226018 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:22.226019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.226644 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.226645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:22.226713 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:22.226718 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:56:22.226746 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 20:56:22.226788 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 20:56:22.227115 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.223911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57454 (local address 127.0.98.3:41389)
0926 20:56:22.224126 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:22.224128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:22.224136 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:56:22.224165 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:22.224166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:22.224175 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:22.224254 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:22.224399 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.225197 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.226112 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.226547 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.226974 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:22.226992 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:22.227011 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 20:56:22.227060 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":49,"threads_started":1}
I20260926 20:56:22.227357 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.223938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42067 (local address 127.0.0.1:54918)
0926 20:56:22.224077 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.224085 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:22.224109 (+ 24us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:22.224451 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:22.224452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:22.224453 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:22.224483 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:22.224484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.225209 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.225210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:22.226376 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:22.226402 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.227169 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.227170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:22.227235 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:22.227239 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:56:22.227288 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 20:56:22.227318 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":24}
I20260926 20:56:22.227435 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.223996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.0.98.1:42067)
0926 20:56:22.224316 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:22.224319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:22.224327 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:56:22.224353 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:22.224354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:22.224365 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:22.224450 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:22.224533 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.225127 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.226501 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.227075 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.227158 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:22.227173 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:22.227297 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 20:56:22.227341 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":70,"threads_started":1}
I20260926 20:56:22.227646 7673 tablet_service.cc:1467] Processing CreateTablet for tablet 8adae7abe8cc4ae78aa3f472d205e5fe (DEFAULT_TABLE table=test-table [id=2eec7dfa89c1418ab91bbc6abdd0a2f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:22.227741 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.223834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46125 (local address 127.0.0.1:36020)
0926 20:56:22.223984 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.223993 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:22.224029 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:22.224237 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:22.224237 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:22.224239 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:22.224297 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:22.224299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.225283 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.225284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:22.226396 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:22.226398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.227525 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.227526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:22.227593 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:22.227599 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:56:22.227645 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 20:56:22.227686 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":51}
I20260926 20:56:22.227744 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 8adae7abe8cc4ae78aa3f472d205e5fe (DEFAULT_TABLE table=test-table [id=2eec7dfa89c1418ab91bbc6abdd0a2f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:22.227779 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8adae7abe8cc4ae78aa3f472d205e5fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:22.227871 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8adae7abe8cc4ae78aa3f472d205e5fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:22.228941 7830 tablet_bootstrap.cc:492] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf: Bootstrap starting.
I20260926 20:56:22.229074 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.224007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36020 (local address 127.0.98.2:46125)
0926 20:56:22.224132 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:22.224134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:22.224142 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:56:22.224168 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:22.224169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:22.224179 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:22.224237 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:22.224361 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.225169 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.226502 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.227354 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.227424 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:22.227435 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:22.228951 (+ 1516us) server_negotiation.cc:299] Negotiation successful
0926 20:56:22.229007 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260926 20:56:22.229261 7537 tablet_service.cc:1467] Processing CreateTablet for tablet 8adae7abe8cc4ae78aa3f472d205e5fe (DEFAULT_TABLE table=test-table [id=2eec7dfa89c1418ab91bbc6abdd0a2f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:22.229373 7537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8adae7abe8cc4ae78aa3f472d205e5fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:22.229490 7830 tablet_bootstrap.cc:654] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:22.230015 7830 tablet_bootstrap.cc:492] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf: No bootstrap required, opened a new log
I20260926 20:56:22.230075 7830 ts_tablet_manager.cc:1397] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:22.230242 7830 raft_consensus.cc:348] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } }
I20260926 20:56:22.230315 7830 raft_consensus.cc:374] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:22.230334 7830 raft_consensus.cc:729] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b23b75bbf57466498a372ea17df29bf, State: Initialized, Role: FOLLOWER
I20260926 20:56:22.230386 7830 consensus_queue.cc:260] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } }
I20260926 20:56:22.230540 7830 ts_tablet_manager.cc:1428] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:22.230815 7834 tablet_bootstrap.cc:492] T 8adae7abe8cc4ae78aa3f472d205e5fe P 2cdb882ced5c4fc383c0ade935a01cab: Bootstrap starting.
I20260926 20:56:22.230989 7831 tablet_bootstrap.cc:492] T 8adae7abe8cc4ae78aa3f472d205e5fe P a812960f384e4f578bbf8392b67794c5: Bootstrap starting.
I20260926 20:56:22.231181 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.230708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.3:36763)
0926 20:56:22.231087 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.231139 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":85,"threads_started":1}
I20260926 20:56:22.231247 7739 heartbeater.cc:503] Master 127.0.98.60:35517 was elected leader, sending a full tablet report...
I20260926 20:56:22.231339 7831 tablet_bootstrap.cc:654] T 8adae7abe8cc4ae78aa3f472d205e5fe P a812960f384e4f578bbf8392b67794c5: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:22.231396 7834 tablet_bootstrap.cc:654] T 8adae7abe8cc4ae78aa3f472d205e5fe P 2cdb882ced5c4fc383c0ade935a01cab: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:22.231810 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.231704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.3:34561)
0926 20:56:22.231747 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.231781 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 20:56:22.232223 7831 tablet_bootstrap.cc:492] T 8adae7abe8cc4ae78aa3f472d205e5fe P a812960f384e4f578bbf8392b67794c5: No bootstrap required, opened a new log
I20260926 20:56:22.232285 7831 ts_tablet_manager.cc:1397] T 8adae7abe8cc4ae78aa3f472d205e5fe P a812960f384e4f578bbf8392b67794c5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:22.232285 7834 tablet_bootstrap.cc:492] T 8adae7abe8cc4ae78aa3f472d205e5fe P 2cdb882ced5c4fc383c0ade935a01cab: No bootstrap required, opened a new log
I20260926 20:56:22.232333 7834 ts_tablet_manager.cc:1397] T 8adae7abe8cc4ae78aa3f472d205e5fe P 2cdb882ced5c4fc383c0ade935a01cab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:56:22.232451 7831 raft_consensus.cc:348] T 8adae7abe8cc4ae78aa3f472d205e5fe P a812960f384e4f578bbf8392b67794c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } }
I20260926 20:56:22.232538 7831 raft_consensus.cc:374] T 8adae7abe8cc4ae78aa3f472d205e5fe P a812960f384e4f578bbf8392b67794c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:22.232493 7834 raft_consensus.cc:348] T 8adae7abe8cc4ae78aa3f472d205e5fe P 2cdb882ced5c4fc383c0ade935a01cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } }
I20260926 20:56:22.232571 7831 raft_consensus.cc:729] T 8adae7abe8cc4ae78aa3f472d205e5fe P a812960f384e4f578bbf8392b67794c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a812960f384e4f578bbf8392b67794c5, State: Initialized, Role: FOLLOWER
I20260926 20:56:22.232578 7834 raft_consensus.cc:374] T 8adae7abe8cc4ae78aa3f472d205e5fe P 2cdb882ced5c4fc383c0ade935a01cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:22.232600 7834 raft_consensus.cc:729] T 8adae7abe8cc4ae78aa3f472d205e5fe P 2cdb882ced5c4fc383c0ade935a01cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cdb882ced5c4fc383c0ade935a01cab, State: Initialized, Role: FOLLOWER
I20260926 20:56:22.232635 7831 consensus_queue.cc:260] T 8adae7abe8cc4ae78aa3f472d205e5fe P a812960f384e4f578bbf8392b67794c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } }
I20260926 20:56:22.232668 7834 consensus_queue.cc:260] T 8adae7abe8cc4ae78aa3f472d205e5fe P 2cdb882ced5c4fc383c0ade935a01cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } }
I20260926 20:56:22.232800 7834 ts_tablet_manager.cc:1428] T 8adae7abe8cc4ae78aa3f472d205e5fe P 2cdb882ced5c4fc383c0ade935a01cab: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:22.232800 7831 ts_tablet_manager.cc:1428] T 8adae7abe8cc4ae78aa3f472d205e5fe P a812960f384e4f578bbf8392b67794c5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:22.233321 7466 heartbeater.cc:503] Master 127.0.98.60:35517 was elected leader, sending a full tablet report...
I20260926 20:56:22.233388 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.232982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.1:43065)
0926 20:56:22.233310 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.233345 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":74,"threads_started":1}
I20260926 20:56:22.233429 7603 heartbeater.cc:503] Master 127.0.98.60:35517 was elected leader, sending a full tablet report...
I20260926 20:56:22.233690 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.233071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.2:51913)
0926 20:56:22.233612 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.233649 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":76,"threads_started":1}
I20260926 20:56:22.233939 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.233691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.1:57179)
0926 20:56:22.233887 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.233914 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260926 20:56:22.234105 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.233946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.2:35247)
0926 20:56:22.234059 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.234083 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 20:56:22.375995 7833 raft_consensus.cc:482] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:56:22.376116 7833 raft_consensus.cc:504] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } }
I20260926 20:56:22.376518 7833 leader_election.cc:288] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a812960f384e4f578bbf8392b67794c5 (127.0.98.1:42067), 2cdb882ced5c4fc383c0ade935a01cab (127.0.98.2:46125)
I20260926 20:56:22.379573 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.376871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40165 (local address 127.0.98.2:46125)
0926 20:56:22.376919 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:22.376922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:22.377083 (+ 161us) server_negotiation.cc:407] Connection header received
0926 20:56:22.377119 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:22.377120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:22.377135 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:22.377201 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:22.377352 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.377881 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.378847 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.379216 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.379270 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:22.379280 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:22.379452 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 20:56:22.379504 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 20:56:22.379587 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.376794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46125 (local address 127.0.98.3:40165)
0926 20:56:22.376996 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.377007 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:22.377070 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:22.377200 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:22.377201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:22.377202 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:22.377281 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:22.377282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.377980 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.377981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:22.378749 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:22.378751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.379300 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.379302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:22.379381 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:22.379384 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:56:22.379427 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 20:56:22.379460 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":3,"thread_start_us":122,"threads_started":1}
I20260926 20:56:22.379639 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.376870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38597 (local address 127.0.98.1:42067)
0926 20:56:22.376923 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:22.376925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:22.376933 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:56:22.376969 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:22.376970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:22.376984 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:22.377069 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:22.377272 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.377817 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.378852 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.379237 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.379327 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:22.379337 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:22.379523 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 20:56:22.379564 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 20:56:22.379859 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.376751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42067 (local address 127.0.98.3:38597)
0926 20:56:22.376829 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.376843 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:22.376907 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:22.377094 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:22.377094 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:22.377097 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:22.377187 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:22.377189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.377905 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.377906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:22.378748 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:22.378750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.379362 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.379363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:22.379431 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:22.379435 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:56:22.379492 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 20:56:22.379539 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":2}
I20260926 20:56:22.379928 7557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8adae7abe8cc4ae78aa3f472d205e5fe" candidate_uuid: "7b23b75bbf57466498a372ea17df29bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cdb882ced5c4fc383c0ade935a01cab" is_pre_election: true
I20260926 20:56:22.380098 7557 raft_consensus.cc:2454] T 8adae7abe8cc4ae78aa3f472d205e5fe P 2cdb882ced5c4fc383c0ade935a01cab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b23b75bbf57466498a372ea17df29bf in term 0.
I20260926 20:56:22.380121 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8adae7abe8cc4ae78aa3f472d205e5fe" candidate_uuid: "7b23b75bbf57466498a372ea17df29bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a812960f384e4f578bbf8392b67794c5" is_pre_election: true
I20260926 20:56:22.380215 7420 raft_consensus.cc:2454] T 8adae7abe8cc4ae78aa3f472d205e5fe P a812960f384e4f578bbf8392b67794c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b23b75bbf57466498a372ea17df29bf in term 0.
I20260926 20:56:22.380339 7627 leader_election.cc:302] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cdb882ced5c4fc383c0ade935a01cab, 7b23b75bbf57466498a372ea17df29bf; no voters:
I20260926 20:56:22.380542 7833 raft_consensus.cc:2781] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:56:22.380580 7833 raft_consensus.cc:482] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:56:22.380595 7833 raft_consensus.cc:3037] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:22.381341 7833 raft_consensus.cc:504] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } }
I20260926 20:56:22.381498 7833 leader_election.cc:288] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [CANDIDATE]: Term 1 election: Requested vote from peers a812960f384e4f578bbf8392b67794c5 (127.0.98.1:42067), 2cdb882ced5c4fc383c0ade935a01cab (127.0.98.2:46125)
I20260926 20:56:22.381665 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8adae7abe8cc4ae78aa3f472d205e5fe" candidate_uuid: "7b23b75bbf57466498a372ea17df29bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a812960f384e4f578bbf8392b67794c5"
I20260926 20:56:22.381742 7420 raft_consensus.cc:3037] T 8adae7abe8cc4ae78aa3f472d205e5fe P a812960f384e4f578bbf8392b67794c5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:22.381695 7557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8adae7abe8cc4ae78aa3f472d205e5fe" candidate_uuid: "7b23b75bbf57466498a372ea17df29bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cdb882ced5c4fc383c0ade935a01cab"
I20260926 20:56:22.381799 7557 raft_consensus.cc:3037] T 8adae7abe8cc4ae78aa3f472d205e5fe P 2cdb882ced5c4fc383c0ade935a01cab [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:22.382367 7420 raft_consensus.cc:2454] T 8adae7abe8cc4ae78aa3f472d205e5fe P a812960f384e4f578bbf8392b67794c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b23b75bbf57466498a372ea17df29bf in term 1.
I20260926 20:56:22.382486 7557 raft_consensus.cc:2454] T 8adae7abe8cc4ae78aa3f472d205e5fe P 2cdb882ced5c4fc383c0ade935a01cab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b23b75bbf57466498a372ea17df29bf in term 1.
I20260926 20:56:22.382536 7626 leader_election.cc:302] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b23b75bbf57466498a372ea17df29bf, a812960f384e4f578bbf8392b67794c5; no voters:
I20260926 20:56:22.382638 7833 raft_consensus.cc:2781] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:22.382782 7833 raft_consensus.cc:686] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [term 1 LEADER]: Becoming Leader. State: Replica: 7b23b75bbf57466498a372ea17df29bf, State: Running, Role: LEADER
I20260926 20:56:22.382884 7833 consensus_queue.cc:237] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } }
I20260926 20:56:22.383203 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.383105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.3:37531)
0926 20:56:22.383138 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.383170 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 20:56:22.383711 7287 catalog_manager.cc:5495] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b23b75bbf57466498a372ea17df29bf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7b23b75bbf57466498a372ea17df29bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b23b75bbf57466498a372ea17df29bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 } health_report { overall_health: UNKNOWN } } }
I20260926 20:56:22.409548 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.407072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.0.98.3:41389)
0926 20:56:22.407135 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:22.407138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:22.407147 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:56:22.407187 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:22.407188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:22.407213 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:22.407295 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:22.407426 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.407909 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.408731 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:22.408941 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:22.409016 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:22.409204 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:22.409309 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:22.409391 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:22.409431 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 20:56:22.409454 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:56:22.410866 7420 raft_consensus.cc:1261] T 8adae7abe8cc4ae78aa3f472d205e5fe P a812960f384e4f578bbf8392b67794c5 [term 1 FOLLOWER]: Refusing update from remote peer 7b23b75bbf57466498a372ea17df29bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:22.410950 7557 raft_consensus.cc:1261] T 8adae7abe8cc4ae78aa3f472d205e5fe P 2cdb882ced5c4fc383c0ade935a01cab [term 1 FOLLOWER]: Refusing update from remote peer 7b23b75bbf57466498a372ea17df29bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:22.411139 7833 consensus_queue.cc:1035] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a812960f384e4f578bbf8392b67794c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:22.411279 7843 consensus_queue.cc:1035] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cdb882ced5c4fc383c0ade935a01cab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:22.412503 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.412239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.2:45807)
0926 20:56:22.412423 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.412468 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260926 20:56:22.412668 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.412345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.1:44831)
0926 20:56:22.412602 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.412636 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233}
I20260926 20:56:22.413609 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.413492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.2:51201)
0926 20:56:22.413525 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.413559 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 20:56:22.413666 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.413544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.3:49573)
0926 20:56:22.413601 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.413633 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 20:56:22.414129 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.414033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.1:47471)
0926 20:56:22.414068 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.414096 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 20:56:22.414487 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.413939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.98.3:57813)
0926 20:56:22.414427 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.414456 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469}
I20260926 20:56:22.414902 7848 mvcc.cc:205] Tried to move back new op lower bound from 7333708523152404480 to 7333708523040927744. Current Snapshot: MvccSnapshot[applied={T|T < 7333708523152404480 or (T in {7333708523152404480})}]
I20260926 20:56:22.415036 7849 mvcc.cc:205] Tried to move back new op lower bound from 7333708523152404480 to 7333708523040927744. Current Snapshot: MvccSnapshot[applied={T|T < 7333708523152404480 or (T in {7333708523152404480})}]
I20260926 20:56:22.417212 7279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:22.417316 7279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:22.418632 7279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79c1cf7b052a412db5bb1a341b1b90d0: Sending DeleteTablet for 3 replicas of tablet 8adae7abe8cc4ae78aa3f472d205e5fe
I20260926 20:56:22.418922 7673 tablet_service.cc:1514] Processing DeleteTablet for tablet 8adae7abe8cc4ae78aa3f472d205e5fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:22 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57454
I20260926 20:56:22.418995 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet 8adae7abe8cc4ae78aa3f472d205e5fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:22 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54918
I20260926 20:56:22.419010 7537 tablet_service.cc:1514] Processing DeleteTablet for tablet 8adae7abe8cc4ae78aa3f472d205e5fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:22 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36020
I20260926 20:56:22.419227 7858 tablet_replica.cc:330] T 8adae7abe8cc4ae78aa3f472d205e5fe P 2cdb882ced5c4fc383c0ade935a01cab: stopping tablet replica
I20260926 20:56:22.419292 7858 raft_consensus.cc:2229] T 8adae7abe8cc4ae78aa3f472d205e5fe P 2cdb882ced5c4fc383c0ade935a01cab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:22.419341 7858 raft_consensus.cc:2258] T 8adae7abe8cc4ae78aa3f472d205e5fe P 2cdb882ced5c4fc383c0ade935a01cab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:22.419479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 7240
I20260926 20:56:22.419523 7856 tablet_replica.cc:330] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf: stopping tablet replica
I20260926 20:56:22.420115 7856 raft_consensus.cc:2229] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:56:22.420285 7856 raft_consensus.cc:2258] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:22.421455 7856 ts_tablet_manager.cc:1905] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:22.423175 7856 ts_tablet_manager.cc:1918] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:22.423300 7856 log.cc:1196] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/8adae7abe8cc4ae78aa3f472d205e5fe
I20260926 20:56:22.423610 7856 ts_tablet_manager.cc:1939] T 8adae7abe8cc4ae78aa3f472d205e5fe P 7b23b75bbf57466498a372ea17df29bf: Deleting consensus metadata
I20260926 20:56:22.424022 7271 catalog_manager.cc:4841] TS 7b23b75bbf57466498a372ea17df29bf (127.0.98.3:41389): tablet 8adae7abe8cc4ae78aa3f472d205e5fe (table test-table [id=2eec7dfa89c1418ab91bbc6abdd0a2f7]) successfully deleted
I20260926 20:56:22.425626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 7474
I20260926 20:56:22.425766 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.425628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42067 (local address 127.0.0.1:54934)
0926 20:56:22.425689 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.425725 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 20:56:22.427940 7271 catalog_manager.cc:4584] TS a812960f384e4f578bbf8392b67794c5 (127.0.98.1:42067): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8adae7abe8cc4ae78aa3f472d205e5fe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42067: connect: Connection refused (error 111)
I20260926 20:56:22.431710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 7610
I20260926 20:56:22.432636 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:22.431603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46125 (local address 127.0.0.1:36022)
0926 20:56:22.432552 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:22.432592 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":921}
I20260926 20:56:22.436884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 6959
I20260926 20:56:22.441767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 7147
2026-09-26T20:56:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4986 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:56:22 dist-test-slave-fl7g krb5kdc[7861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:56:22 dist-test-slave-fl7g krb5kdc[7861](info): set up 2 sockets
Sep 26 20:56:22 dist-test-slave-fl7g krb5kdc[7861](info): commencing operation
krb5kdc: starting...
W20260926 20:56:23.485070 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:56:23 dist-test-slave-fl7g krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:56:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 20:56:23.611639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45723
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:23.693781 7874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:23.694028 7874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:23.694053 7874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:23.695312 7874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:23.695363 7874 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:56:23.695377 7874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:23.695391 7874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:23.695403 7874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:23.695415 7874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:23.696997 7874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45723
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:23.697224 7874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:23.697412 7874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:23.702642 7882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:23.702711 7880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:23.702868 7879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:23.703402 7874 server_base.cc:1034] running on GCE node
I20260926 20:56:23.703667 7874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:23.703974 7874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:23.705171 7874 hybrid_clock.cc:648] HybridClock initialized: now 1790456183705153 us; error 32 us; skew 500 ppm
Sep 26 20:56:23 dist-test-slave-fl7g krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456183, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:23.707633 7874 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 20:56:23.708086 7874 webserver.cc:460] Webserver started at http://127.0.98.62:44743/ using document root <none> and password file <none>
I20260926 20:56:23.708328 7874 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:23.708392 7874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:23.708570 7874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:23.709658 7874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "b8c39014a0ad4bf69adf32457e6906ca"
format_stamp: "Formatted at 2026-09-26 20:56:23 on dist-test-slave-fl7g"
server_key: "403ea968971631291c089d45ab5911df"
server_key_iv: "c16bf4697c8e07b94aa9ec2ff65e4a26"
server_key_version: "encryptionkey@0"
I20260926 20:56:23.710005 7874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "b8c39014a0ad4bf69adf32457e6906ca"
format_stamp: "Formatted at 2026-09-26 20:56:23 on dist-test-slave-fl7g"
server_key: "403ea968971631291c089d45ab5911df"
server_key_iv: "c16bf4697c8e07b94aa9ec2ff65e4a26"
server_key_version: "encryptionkey@0"
I20260926 20:56:23.711350 7874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:56:23.712044 7889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:23.712214 7874 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:23.712309 7874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "b8c39014a0ad4bf69adf32457e6906ca"
format_stamp: "Formatted at 2026-09-26 20:56:23 on dist-test-slave-fl7g"
server_key: "403ea968971631291c089d45ab5911df"
server_key_iv: "c16bf4697c8e07b94aa9ec2ff65e4a26"
server_key_version: "encryptionkey@0"
I20260926 20:56:23.712387 7874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:23.723330 7874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:23.725713 7874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:23.725888 7874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:23.730732 7874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45723
I20260926 20:56:23.730774 7951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45723 every 8 connection(s)
I20260926 20:56:23.731180 7874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:23.731797 7952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:23.734117 7952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca: Bootstrap starting.
I20260926 20:56:23.734717 7952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:23.735009 7952 log.cc:824] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:23.735756 7952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca: No bootstrap required, opened a new log
I20260926 20:56:23.736676 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 7874
I20260926 20:56:23.736754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:56:23.736935 392 external_mini_cluster.cc:1428] Setting key 6a148342bd3c1b033622b76f81733bf5
I20260926 20:56:23.737671 7952 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c39014a0ad4bf69adf32457e6906ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45723 } }
I20260926 20:56:23.737866 7952 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:23.737919 7952 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c39014a0ad4bf69adf32457e6906ca, State: Initialized, Role: FOLLOWER
I20260926 20:56:23.738059 7952 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c39014a0ad4bf69adf32457e6906ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45723 } }
I20260926 20:56:23.738126 7952 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:23.738181 7952 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:23.738231 7952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:23.739004 7952 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c39014a0ad4bf69adf32457e6906ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45723 } }
I20260926 20:56:23.739151 7952 leader_election.cc:302] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8c39014a0ad4bf69adf32457e6906ca; no voters:
I20260926 20:56:23.739321 7952 leader_election.cc:288] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:23.739352 7957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:23.739467 7957 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca [term 1 LEADER]: Becoming Leader. State: Replica: b8c39014a0ad4bf69adf32457e6906ca, State: Running, Role: LEADER
I20260926 20:56:23.739579 7952 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:23.739605 7957 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c39014a0ad4bf69adf32457e6906ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45723 } }
I20260926 20:56:23.740013 7957 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8c39014a0ad4bf69adf32457e6906ca. Latest consensus state: current_term: 1 leader_uuid: "b8c39014a0ad4bf69adf32457e6906ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c39014a0ad4bf69adf32457e6906ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45723 } } }
I20260926 20:56:23.740272 7957 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:23.740193 7959 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8c39014a0ad4bf69adf32457e6906ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c39014a0ad4bf69adf32457e6906ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45723 } } }
I20260926 20:56:23.740465 7959 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:23.740869 7966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:23.741425 7966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:23.743260 7966 catalog_manager.cc:1347] Generated new cluster ID: 072b5c908f164a5d913472d2cf99e551
I20260926 20:56:23.743316 7966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:56:23 dist-test-slave-fl7g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:56:23.748728 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:23.737757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.0.98.62:45723)
0926 20:56:23.737914 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:23.737924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:23.737956 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:56:23.738087 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:23.738095 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:23.738174 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:23.738322 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:23.739117 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:23.740523 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:23.741575 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:23.742173 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:23.742302 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:23.745120 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:23.745124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:23.745124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:23.745142 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:23.747060 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:23.747479 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:23.747482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:23.747482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:23.747597 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:23.747802 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:23.747803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:23.747803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:23.748090 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:23.748241 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:23.748357 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 20:56:23.748509 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260926 20:56:23.777225 7966 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:23.777807 7966 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:23.781884 7966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca: Generated new TSK 0
I20260926 20:56:23.782111 7966 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 20:56:23.812583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45723
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:23.898123 7978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:23.898351 7978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:23.898378 7978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:23.899659 7978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:23.899709 7978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:23.899731 7978 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:56:23.899750 7978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:56:23.901341 7978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:23.901624 7978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:23.901868 7978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:23.907284 7986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:23.907308 7983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:23.907292 7984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:23.907668 7978 server_base.cc:1034] running on GCE node
I20260926 20:56:23.907914 7978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:23.908180 7978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:23.909346 7978 hybrid_clock.cc:648] HybridClock initialized: now 1790456183909323 us; error 52 us; skew 500 ppm
Sep 26 20:56:23 dist-test-slave-fl7g krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456183, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:23.912173 7978 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 20:56:23.912732 7978 webserver.cc:460] Webserver started at http://127.0.98.1:38551/ using document root <none> and password file <none>
I20260926 20:56:23.912971 7978 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:23.913039 7978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:23.913197 7978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:23.914294 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "0552e62e93104b9488582a9b72bc0c35"
format_stamp: "Formatted at 2026-09-26 20:56:23 on dist-test-slave-fl7g"
server_key: "8cc976a72222fc15106cedb5ff8dfe73"
server_key_iv: "c8776fa096fcecdcb4d6639779101ea3"
server_key_version: "encryptionkey@0"
I20260926 20:56:23.914659 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0552e62e93104b9488582a9b72bc0c35"
format_stamp: "Formatted at 2026-09-26 20:56:23 on dist-test-slave-fl7g"
server_key: "8cc976a72222fc15106cedb5ff8dfe73"
server_key_iv: "c8776fa096fcecdcb4d6639779101ea3"
server_key_version: "encryptionkey@0"
I20260926 20:56:23.916134 7978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:56:23.916970 7993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:23.917124 7978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:23.917212 7978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "0552e62e93104b9488582a9b72bc0c35"
format_stamp: "Formatted at 2026-09-26 20:56:23 on dist-test-slave-fl7g"
server_key: "8cc976a72222fc15106cedb5ff8dfe73"
server_key_iv: "c8776fa096fcecdcb4d6639779101ea3"
server_key_version: "encryptionkey@0"
I20260926 20:56:23.917269 7978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:23.929401 7978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:23.931883 7978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:23.932104 7978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:23.932722 7978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:23.932781 7978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:23.932807 7978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:23.932850 7978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:23.939949 7978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39217
I20260926 20:56:23.940021 8108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39217 every 8 connection(s)
Sep 26 20:56:23 dist-test-slave-fl7g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456183, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:56:23.940560 7978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:56:23.943609 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:23.933346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48823 (local address 127.0.98.62:45723)
0926 20:56:23.933427 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:23.933429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:23.933630 (+ 201us) server_negotiation.cc:407] Connection header received
0926 20:56:23.934181 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:23.934182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:23.934198 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:23.934256 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:23.935194 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:23.936080 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:23.937854 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:23.938066 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:23.938202 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:23.941060 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:23.941062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:23.941062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:23.941070 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:23.942378 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:23.942785 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:23.942786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:23.942786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:23.942837 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:23.943054 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:23.943055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:23.943055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:23.943161 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:23.943212 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:23.943439 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 20:56:23.943510 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:56:23.943748 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:23.933318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45723 (local address 127.0.98.1:48823)
0926 20:56:23.933495 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:23.933539 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:23.934074 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:23.934337 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:23.934342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:23.934749 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:23.935072 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:23.935080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:23.936215 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:23.936216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:23.937752 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:23.937756 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:23.938221 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:23.938222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:23.938384 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:23.939101 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:23.939119 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:23.940956 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:23.942459 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:23.942465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:23.942467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:23.942715 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:23.942891 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:23.942891 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:23.942891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:23.943001 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:23.943212 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:23.943216 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:23.943337 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 20:56:23.943426 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:56:23.943551 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":105,"threads_started":1}
I20260926 20:56:23.944249 8109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45723
I20260926 20:56:23.944339 8109 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:23.944561 8109 heartbeater.cc:511] Master 127.0.98.62:45723 requested a full tablet report, sending...
I20260926 20:56:23.945129 7905 ts_manager.cc:194] Registered new tserver with Master: 0552e62e93104b9488582a9b72bc0c35 (127.0.98.1:39217)
I20260926 20:56:23.945994 7905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48823
I20260926 20:56:23.947868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 7978
I20260926 20:56:23.947984 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:56:23.948191 392 external_mini_cluster.cc:1428] Setting key a6e35c8d0808d63f3a46c79fd5a7d459
I20260926 20:56:23.951390 8115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:23.945367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45723 (local address 127.0.98.1:49981)
0926 20:56:23.945519 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:23.945530 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:23.945583 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:23.945694 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:23.945694 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:23.945852 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:23.945905 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:23.945907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:23.946768 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:23.946769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:23.947312 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:23.947313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:23.947650 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:23.947651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:23.947717 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:23.948237 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:23.948248 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:23.948686 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:23.950335 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:23.950336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:23.950337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:23.950530 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:23.950770 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:23.950771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:23.950771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:23.950836 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:23.951128 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:23.951129 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:23.951169 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 20:56:23.951225 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:56:23.951309 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260926 20:56:23.951517 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:23.945422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49981 (local address 127.0.98.62:45723)
0926 20:56:23.945466 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:23.945469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:23.945590 (+ 121us) server_negotiation.cc:407] Connection header received
0926 20:56:23.945615 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:23.945616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:23.945626 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:23.945691 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:23.945963 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:23.946646 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:23.947392 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:23.947584 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:23.947642 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:23.948757 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:23.948758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:23.948758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:23.948765 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:23.950237 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:23.950599 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:23.950600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:23.950601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:23.950667 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:23.950907 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:23.950908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:23.950908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:23.951055 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:23.951129 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:23.951270 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 20:56:23.951359 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 20:56:23.979089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45723
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:24.063670 8118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:24.063901 8118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:24.063928 8118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:24.065191 8118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:24.065248 8118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:24.065270 8118 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:56:24.065289 8118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:56:24.066879 8118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:24.067157 8118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:24.067399 8118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:24.072589 8124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:24.072641 8126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:24.072585 8123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:24.072806 8118 server_base.cc:1034] running on GCE node
I20260926 20:56:24.073041 8118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:24.073285 8118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:24.074481 8118 hybrid_clock.cc:648] HybridClock initialized: now 1790456184074468 us; error 36 us; skew 500 ppm
Sep 26 20:56:24 dist-test-slave-fl7g krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456184, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:24.077041 8118 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 20:56:24.077627 8118 webserver.cc:460] Webserver started at http://127.0.98.2:37621/ using document root <none> and password file <none>
I20260926 20:56:24.077867 8118 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:24.077932 8118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:24.078081 8118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:24.079162 8118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "f79ca367d62b46bdb32256e3cfcf5742"
format_stamp: "Formatted at 2026-09-26 20:56:24 on dist-test-slave-fl7g"
server_key: "cb95980cafc30cef2b557316ba9d6bfc"
server_key_iv: "1bbd771cc00646c871eb37e463ba8a76"
server_key_version: "encryptionkey@0"
I20260926 20:56:24.079533 8118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f79ca367d62b46bdb32256e3cfcf5742"
format_stamp: "Formatted at 2026-09-26 20:56:24 on dist-test-slave-fl7g"
server_key: "cb95980cafc30cef2b557316ba9d6bfc"
server_key_iv: "1bbd771cc00646c871eb37e463ba8a76"
server_key_version: "encryptionkey@0"
I20260926 20:56:24.081082 8118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:56:24.081982 8133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:24.082247 8118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:24.082370 8118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "f79ca367d62b46bdb32256e3cfcf5742"
format_stamp: "Formatted at 2026-09-26 20:56:24 on dist-test-slave-fl7g"
server_key: "cb95980cafc30cef2b557316ba9d6bfc"
server_key_iv: "1bbd771cc00646c871eb37e463ba8a76"
server_key_version: "encryptionkey@0"
I20260926 20:56:24.082486 8118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:24.097332 8118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:24.099661 8118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:24.099846 8118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:24.100482 8118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:24.100539 8118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:24.100592 8118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:24.100620 8118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:56:24 dist-test-slave-fl7g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456184, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:56:24.108158 8118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35471
I20260926 20:56:24.108592 8248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35471 every 8 connection(s)
I20260926 20:56:24.108686 8118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:56:24.110747 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.100982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51645 (local address 127.0.98.62:45723)
0926 20:56:24.101060 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.101063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.101367 (+ 304us) server_negotiation.cc:407] Connection header received
0926 20:56:24.102112 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.102113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.102127 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.102196 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:24.103214 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.103786 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.105033 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.105247 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.105323 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.107611 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:24.107612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:24.107612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:24.107621 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:24.109355 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:24.109781 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:24.109782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:24.109783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:24.109848 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:24.110106 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:24.110107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:24.110108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:24.110226 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:24.110298 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.110566 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.110658 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:56:24.110945 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.100971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45723 (local address 127.0.98.2:51645)
0926 20:56:24.101181 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:24.101240 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:24.101985 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:24.102292 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:24.102298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:24.102727 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:24.103079 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.103088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.103894 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.103896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.104937 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.104940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.105343 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.105344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.105478 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.106039 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:24.106048 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:24.107509 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:24.109448 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:24.109454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:24.109457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:24.109691 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:24.109913 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:24.109914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:24.109915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:24.110034 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:24.110289 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:24.110295 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:24.110436 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 20:56:24.110543 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 20:56:24.110709 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":103,"threads_started":1}
I20260926 20:56:24.111423 8249 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45723
I20260926 20:56:24.111500 8249 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:24.111703 8249 heartbeater.cc:511] Master 127.0.98.62:45723 requested a full tablet report, sending...
I20260926 20:56:24.112080 7906 ts_manager.cc:194] Registered new tserver with Master: f79ca367d62b46bdb32256e3cfcf5742 (127.0.98.2:35471)
I20260926 20:56:24.112627 7906 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51645
I20260926 20:56:24.114506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 8118
I20260926 20:56:24.114646 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:56:24.114840 392 external_mini_cluster.cc:1428] Setting key e1bfb22685e926c5017f593c90b741d6
I20260926 20:56:24.118439 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.112469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45723 (local address 127.0.98.2:52719)
0926 20:56:24.112596 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:24.112606 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:24.112686 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:24.112815 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:24.112816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:24.113022 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:24.113082 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.113084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.113717 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.113718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.114244 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.114245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.114658 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.114659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.114724 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.115206 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:24.115211 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:24.115577 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:24.117463 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:24.117464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:24.117465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:24.117631 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:24.117826 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:24.117826 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:24.117827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:24.117875 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:24.118204 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:24.118205 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:24.118241 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 20:56:24.118288 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 20:56:24.118365 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260926 20:56:24.118542 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.112511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52719 (local address 127.0.98.62:45723)
0926 20:56:24.112546 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.112548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.112670 (+ 122us) server_negotiation.cc:407] Connection header received
0926 20:56:24.112745 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.112746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.112756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.112811 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:24.113157 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.113625 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.114349 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.114556 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.114633 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.115664 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:24.115665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:24.115666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:24.115671 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:24.117368 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:24.117698 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:24.117699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:24.117699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:24.117763 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:24.117931 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:24.117932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:24.117932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:24.118136 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:24.118202 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.118324 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.118405 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 20:56:24.143270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45723
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:24.224918 8258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:24.225136 8258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:24.225162 8258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:24.226289 8258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:24.226325 8258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:24.226346 8258 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:56:24.226363 8258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:56:24.227774 8258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:24.227974 8258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:24.228150 8258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:24.233325 8266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:24.233314 8264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:24.233314 8263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:24.233609 8258 server_base.cc:1034] running on GCE node
I20260926 20:56:24.233862 8258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:24.234077 8258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:24.235260 8258 hybrid_clock.cc:648] HybridClock initialized: now 1790456184235247 us; error 37 us; skew 500 ppm
Sep 26 20:56:24 dist-test-slave-fl7g krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456184, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:24.238019 8258 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 20:56:24.238516 8258 webserver.cc:460] Webserver started at http://127.0.98.3:45015/ using document root <none> and password file <none>
I20260926 20:56:24.238734 8258 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:24.238790 8258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:24.238900 8258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:24.240123 8258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "78209d3846f8467bb3fd656dffeb72c0"
format_stamp: "Formatted at 2026-09-26 20:56:24 on dist-test-slave-fl7g"
server_key: "6c878e5d656385661182d4919aab5f56"
server_key_iv: "228cedafe097d43bf290dc774f6dfcd2"
server_key_version: "encryptionkey@0"
I20260926 20:56:24.240731 8258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "78209d3846f8467bb3fd656dffeb72c0"
format_stamp: "Formatted at 2026-09-26 20:56:24 on dist-test-slave-fl7g"
server_key: "6c878e5d656385661182d4919aab5f56"
server_key_iv: "228cedafe097d43bf290dc774f6dfcd2"
server_key_version: "encryptionkey@0"
I20260926 20:56:24.242875 8258 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260926 20:56:24.243863 8273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:24.244061 8258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:24.244160 8258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "78209d3846f8467bb3fd656dffeb72c0"
format_stamp: "Formatted at 2026-09-26 20:56:24 on dist-test-slave-fl7g"
server_key: "6c878e5d656385661182d4919aab5f56"
server_key_iv: "228cedafe097d43bf290dc774f6dfcd2"
server_key_version: "encryptionkey@0"
I20260926 20:56:24.244220 8258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:24.257401 8258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:24.259814 8258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:24.259935 8258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:24.260479 8258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:24.260525 8258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:24.260550 8258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:24.260600 8258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:24.267724 8258 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38091
I20260926 20:56:24.267813 8388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38091 every 8 connection(s)
Sep 26 20:56:24 dist-test-slave-fl7g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456184, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:56:24.268222 8258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:56:24.268460 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 8258
I20260926 20:56:24.268615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:56:24.268872 392 external_mini_cluster.cc:1428] Setting key 46ada4774f49af4c3ba8febbb081757c
I20260926 20:56:24.271937 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.261034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59175 (local address 127.0.98.62:45723)
0926 20:56:24.261077 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.261079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.261427 (+ 348us) server_negotiation.cc:407] Connection header received
0926 20:56:24.261984 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.261985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.261997 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.262048 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:24.262969 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.263813 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.265469 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.265796 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.265881 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.268787 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:24.268789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:24.268790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:24.268799 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:24.270603 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:24.270974 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:24.270975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:24.270975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:24.271042 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:24.271266 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:24.271266 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:24.271266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:24.271374 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:24.271423 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.271762 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.271851 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:56:24.272051 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.261034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45723 (local address 127.0.98.3:59175)
0926 20:56:24.261285 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:24.261325 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:24.261873 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:24.262120 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:24.262125 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:24.262480 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:24.262812 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.262818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.263948 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.263949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.265361 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.265365 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.265899 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.265900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.266010 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.266664 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:24.266675 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:24.268671 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:24.270675 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:24.270681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:24.270683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:24.270909 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:24.271106 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:24.271106 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:24.271107 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:24.271213 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:24.271466 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:24.271472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:24.271608 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 20:56:24.271712 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 20:56:24.271847 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":106,"threads_started":1}
I20260926 20:56:24.272547 8389 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45723
I20260926 20:56:24.272635 8389 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:24.272852 8389 heartbeater.cc:511] Master 127.0.98.62:45723 requested a full tablet report, sending...
I20260926 20:56:24.273304 7905 ts_manager.cc:194] Registered new tserver with Master: 78209d3846f8467bb3fd656dffeb72c0 (127.0.98.3:38091)
I20260926 20:56:24.273829 7905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59175
I20260926 20:56:24.279371 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.273677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42245 (local address 127.0.98.62:45723)
0926 20:56:24.273731 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.273733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.273801 (+ 68us) server_negotiation.cc:407] Connection header received
0926 20:56:24.273865 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.273866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.273873 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.273935 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:24.274322 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.274794 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.275503 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.275701 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.275781 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.276991 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:24.276992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:24.276992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:24.276999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:24.278217 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:24.278609 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:24.278610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:24.278611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:24.278667 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:24.278854 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:24.278855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:24.278855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:24.278966 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:24.279034 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.279194 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.279274 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:56:24.279385 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.273595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45723 (local address 127.0.98.3:42245)
0926 20:56:24.273736 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:24.273748 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:24.273810 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:24.273953 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:24.273954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:24.274175 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:24.274245 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.274247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.274888 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.274889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.275404 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.275406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.275782 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.275782 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.275847 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.276278 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:24.276282 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:24.276889 (+ 607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:24.278338 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:24.278339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:24.278340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:24.278527 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:24.278728 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:24.278728 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:24.278729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:24.278793 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:24.279045 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:24.279046 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:24.279092 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 20:56:24.279169 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:56:24.279276 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260926 20:56:24.280611 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:56:24 dist-test-slave-fl7g krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456184, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:56:24 dist-test-slave-fl7g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:24 dist-test-slave-fl7g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:56:24.302697 7956 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:24.296663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60462 (local address 127.0.98.62:45723)
0926 20:56:24.296727 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.296729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.296742 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:56:24.296838 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.296839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.296851 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.296923 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:24.297525 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.298026 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.298795 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.298997 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.299088 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.302582 (+ 3494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60462: BlockingRecv error: recv got EOF from 127.0.0.1:60462 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 20:56:24 dist-test-slave-fl7g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456184, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:56:24.310734 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.303587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.0.98.62:45723)
0926 20:56:24.303632 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.303634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.303644 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:56:24.303709 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.303710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.303720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.303788 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:24.304283 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.304859 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.305554 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.305753 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.305838 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.308150 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:24.308152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:24.308152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:24.308160 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:24.309576 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:24.309949 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:24.309950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:24.309951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:24.310010 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:24.310204 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:24.310205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:24.310205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:24.310327 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:24.310432 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.310533 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.310622 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:56:24.312637 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:56:24.314293 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:56:24.320894 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.316791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51548 (local address 127.0.98.2:35471)
0926 20:56:24.316957 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.316980 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.316992 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:56:24.317041 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.317043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.317061 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.317143 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:24.317414 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.318220 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.319512 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.320284 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.320333 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.320497 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.320759 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.320800 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260926 20:56:24.320885 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.316669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35471 (local address 127.0.0.1:51548)
0926 20:56:24.316790 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:24.316833 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:24.316896 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:24.317163 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:24.317166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:24.317168 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:24.317256 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.317259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.318309 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.318310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.319408 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.319410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.320558 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.320559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.320639 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.320653 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 20:56:24.320722 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:56:24.320779 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":33,"thread_start_us":74,"threads_started":1}
I20260926 20:56:24.320964 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.316674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38091 (local address 127.0.0.1:43410)
0926 20:56:24.317013 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:24.317023 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:24.317075 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:24.317557 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:24.317557 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:24.317559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:24.317607 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.317608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.318501 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.318502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.319659 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.319661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.320565 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.320566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.320691 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.320695 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:56:24.320753 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:56:24.320794 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260926 20:56:24.321259 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.317040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.0.98.3:38091)
0926 20:56:24.317332 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.317337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.317347 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:56:24.317385 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.317387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.317403 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.317479 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:24.317667 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.318405 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.319732 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.320489 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.320563 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.320641 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.320931 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.321166 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":85,"threads_started":1}
I20260926 20:56:24.321858 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 480166ab0e3f40c3a9d65d8c594110ec (DEFAULT_TABLE table=test-table [id=8c6764d3f85c4305b07556c5f55a02ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:24.321858 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 480166ab0e3f40c3a9d65d8c594110ec (DEFAULT_TABLE table=test-table [id=8c6764d3f85c4305b07556c5f55a02ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:24.322170 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.316894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39217 (local address 127.0.0.1:35714)
0926 20:56:24.317708 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:24.317718 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:24.317757 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:24.318584 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:24.318585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:24.318587 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:24.318623 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.318625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.319887 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.319888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.320814 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.320816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.321912 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.321913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.321997 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.322002 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:56:24.322056 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:56:24.322102 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"thread_start_us":80,"threads_started":1}
I20260926 20:56:24.322253 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 480166ab0e3f40c3a9d65d8c594110ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:24.322253 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 480166ab0e3f40c3a9d65d8c594110ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:24.322252 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.317339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35714 (local address 127.0.98.1:39217)
0926 20:56:24.318387 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.318391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.318400 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:56:24.318434 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.318436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.318451 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.318523 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:24.318708 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.319828 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.320926 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.321811 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.321900 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.321993 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.322101 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.322141 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1014,"thread_start_us":91,"threads_started":1}
I20260926 20:56:24.322923 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 480166ab0e3f40c3a9d65d8c594110ec (DEFAULT_TABLE table=test-table [id=8c6764d3f85c4305b07556c5f55a02ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:24.323254 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 480166ab0e3f40c3a9d65d8c594110ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:24.326355 8417 tablet_bootstrap.cc:492] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0: Bootstrap starting.
I20260926 20:56:24.327068 8419 tablet_bootstrap.cc:492] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742: Bootstrap starting.
I20260926 20:56:24.327113 8418 tablet_bootstrap.cc:492] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35: Bootstrap starting.
I20260926 20:56:24.327178 8417 tablet_bootstrap.cc:654] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:24.327566 8417 log.cc:824] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:24.327672 8419 tablet_bootstrap.cc:654] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:24.327746 8418 tablet_bootstrap.cc:654] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:24.328016 8419 log.cc:824] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:24.328090 8418 log.cc:824] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:24.328519 8417 tablet_bootstrap.cc:492] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0: No bootstrap required, opened a new log
I20260926 20:56:24.328596 8417 ts_tablet_manager.cc:1397] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260926 20:56:24.328819 8418 tablet_bootstrap.cc:492] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35: No bootstrap required, opened a new log
I20260926 20:56:24.328821 8419 tablet_bootstrap.cc:492] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742: No bootstrap required, opened a new log
I20260926 20:56:24.328866 8418 ts_tablet_manager.cc:1397] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:56:24.328876 8419 ts_tablet_manager.cc:1397] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:56:24.330507 8418 raft_consensus.cc:348] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } }
I20260926 20:56:24.330643 8418 raft_consensus.cc:374] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:24.330664 8418 raft_consensus.cc:729] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0552e62e93104b9488582a9b72bc0c35, State: Initialized, Role: FOLLOWER
I20260926 20:56:24.330752 8418 consensus_queue.cc:260] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } }
I20260926 20:56:24.330744 8417 raft_consensus.cc:348] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } }
I20260926 20:56:24.330904 8417 raft_consensus.cc:374] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:24.330941 8417 raft_consensus.cc:729] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78209d3846f8467bb3fd656dffeb72c0, State: Initialized, Role: FOLLOWER
I20260926 20:56:24.330919 8419 raft_consensus.cc:348] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } }
I20260926 20:56:24.331053 8419 raft_consensus.cc:374] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:24.331034 8418 ts_tablet_manager.cc:1428] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:56:24.331045 8109 heartbeater.cc:503] Master 127.0.98.62:45723 was elected leader, sending a full tablet report...
I20260926 20:56:24.331085 8419 raft_consensus.cc:729] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f79ca367d62b46bdb32256e3cfcf5742, State: Initialized, Role: FOLLOWER
I20260926 20:56:24.331072 8417 consensus_queue.cc:260] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } }
I20260926 20:56:24.331188 8419 consensus_queue.cc:260] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } }
I20260926 20:56:24.331350 8417 ts_tablet_manager.cc:1428] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 20:56:24.331393 8389 heartbeater.cc:503] Master 127.0.98.62:45723 was elected leader, sending a full tablet report...
I20260926 20:56:24.331465 8419 ts_tablet_manager.cc:1428] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:56:24.331605 8249 heartbeater.cc:503] Master 127.0.98.62:45723 was elected leader, sending a full tablet report...
W20260926 20:56:24.359371 8250 tablet.cc:2369] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:56:24.366598 8425 raft_consensus.cc:482] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:56:24.366731 8425 raft_consensus.cc:504] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } }
I20260926 20:56:24.367039 8425 leader_election.cc:288] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78209d3846f8467bb3fd656dffeb72c0 (127.0.98.3:38091), 0552e62e93104b9488582a9b72bc0c35 (127.0.98.1:39217)
I20260926 20:56:24.371263 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.367394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53877 (local address 127.0.98.1:39217)
0926 20:56:24.367461 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.367463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.367472 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:56:24.367501 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.367502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.367515 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.367604 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:24.367812 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.368611 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.370251 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.370845 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.370917 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.370928 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.371147 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.371187 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:56:24.371263 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.367355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41557 (local address 127.0.98.3:38091)
0926 20:56:24.367405 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.367406 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.367598 (+ 192us) server_negotiation.cc:407] Connection header received
0926 20:56:24.367704 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.367705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.367719 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.367774 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:24.367940 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.368651 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.370251 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.370829 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.370917 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.370928 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.371145 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.371187 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 20:56:24.371317 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.367276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38091 (local address 127.0.98.2:41557)
0926 20:56:24.367500 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:24.367511 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:24.367607 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:24.367781 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:24.367782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:24.367784 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:24.367856 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.367858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.368737 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.368738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.370127 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.370131 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.370942 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.370942 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.371055 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.371061 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:56:24.371129 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:56:24.371174 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":84,"thread_start_us":122,"threads_started":1}
I20260926 20:56:24.371322 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.367276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39217 (local address 127.0.98.2:53877)
0926 20:56:24.367349 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:24.367362 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:24.367440 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:24.367642 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:24.367644 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:24.367646 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:24.367723 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.367725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.368737 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.368738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.370133 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.370135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.370941 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.370942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.371055 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.371060 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:56:24.371129 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:56:24.371180 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":136}
I20260926 20:56:24.371785 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480166ab0e3f40c3a9d65d8c594110ec" candidate_uuid: "f79ca367d62b46bdb32256e3cfcf5742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0552e62e93104b9488582a9b72bc0c35" is_pre_election: true
I20260926 20:56:24.371785 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480166ab0e3f40c3a9d65d8c594110ec" candidate_uuid: "f79ca367d62b46bdb32256e3cfcf5742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78209d3846f8467bb3fd656dffeb72c0" is_pre_election: true
I20260926 20:56:24.371959 8063 raft_consensus.cc:2454] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f79ca367d62b46bdb32256e3cfcf5742 in term 0.
I20260926 20:56:24.371959 8343 raft_consensus.cc:2454] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f79ca367d62b46bdb32256e3cfcf5742 in term 0.
I20260926 20:56:24.372212 8137 leader_election.cc:302] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0552e62e93104b9488582a9b72bc0c35, f79ca367d62b46bdb32256e3cfcf5742; no voters:
I20260926 20:56:24.372479 8425 raft_consensus.cc:2781] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:56:24.372575 8425 raft_consensus.cc:482] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:56:24.372611 8425 raft_consensus.cc:3037] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:24.373739 8425 raft_consensus.cc:504] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } }
I20260926 20:56:24.373921 8425 leader_election.cc:288] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [CANDIDATE]: Term 1 election: Requested vote from peers 78209d3846f8467bb3fd656dffeb72c0 (127.0.98.3:38091), 0552e62e93104b9488582a9b72bc0c35 (127.0.98.1:39217)
I20260926 20:56:24.374091 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480166ab0e3f40c3a9d65d8c594110ec" candidate_uuid: "f79ca367d62b46bdb32256e3cfcf5742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0552e62e93104b9488582a9b72bc0c35"
I20260926 20:56:24.374123 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480166ab0e3f40c3a9d65d8c594110ec" candidate_uuid: "f79ca367d62b46bdb32256e3cfcf5742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78209d3846f8467bb3fd656dffeb72c0"
I20260926 20:56:24.374187 8063 raft_consensus.cc:3037] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:24.374200 8343 raft_consensus.cc:3037] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:24.375066 8343 raft_consensus.cc:2454] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f79ca367d62b46bdb32256e3cfcf5742 in term 1.
I20260926 20:56:24.375066 8063 raft_consensus.cc:2454] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f79ca367d62b46bdb32256e3cfcf5742 in term 1.
I20260926 20:56:24.375228 8137 leader_election.cc:302] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0552e62e93104b9488582a9b72bc0c35, f79ca367d62b46bdb32256e3cfcf5742; no voters:
I20260926 20:56:24.375324 8425 raft_consensus.cc:2781] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:24.375468 8425 raft_consensus.cc:686] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [term 1 LEADER]: Becoming Leader. State: Replica: f79ca367d62b46bdb32256e3cfcf5742, State: Running, Role: LEADER
I20260926 20:56:24.375572 8425 consensus_queue.cc:237] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } }
I20260926 20:56:24.376649 7906 catalog_manager.cc:5495] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 reported cstate change: term changed from 0 to 1, leader changed from <none> to f79ca367d62b46bdb32256e3cfcf5742 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f79ca367d62b46bdb32256e3cfcf5742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } health_report { overall_health: UNKNOWN } } }
I20260926 20:56:24.385085 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.382472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51564 (local address 127.0.98.2:35471)
0926 20:56:24.382522 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.382524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.382534 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:56:24.382607 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.382608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.382630 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.382711 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:24.382889 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.383397 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.384196 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.384383 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.384505 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.384702 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:24.384821 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:24.384938 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.384959 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.384981 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 20:56:24.387043 8343 raft_consensus.cc:1261] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0 [term 1 FOLLOWER]: Refusing update from remote peer f79ca367d62b46bdb32256e3cfcf5742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:24.387409 8063 raft_consensus.cc:1261] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35 [term 1 FOLLOWER]: Refusing update from remote peer f79ca367d62b46bdb32256e3cfcf5742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:24.387527 8427 consensus_queue.cc:1035] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:24.387719 8425 consensus_queue.cc:1035] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:24.390164 8430 mvcc.cc:205] Tried to move back new op lower bound from 7333708531246358528 to 7333708531203059712. Current Snapshot: MvccSnapshot[applied={T|T < 7333708531246358528}]
I20260926 20:56:24.390563 8431 mvcc.cc:205] Tried to move back new op lower bound from 7333708531246358528 to 7333708531203059712. Current Snapshot: MvccSnapshot[applied={T|T < 7333708531246358528}]
I20260926 20:56:24.390821 8432 mvcc.cc:205] Tried to move back new op lower bound from 7333708531246358528 to 7333708531203059712. Current Snapshot: MvccSnapshot[applied={T|T < 7333708531246358528}]
I20260926 20:56:24.392560 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:24.392683 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:24.393517 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca: Sending DeleteTablet for 3 replicas of tablet 480166ab0e3f40c3a9d65d8c594110ec
I20260926 20:56:24.393883 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 480166ab0e3f40c3a9d65d8c594110ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43410
I20260926 20:56:24.393959 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 480166ab0e3f40c3a9d65d8c594110ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35714
I20260926 20:56:24.394199 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet 480166ab0e3f40c3a9d65d8c594110ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51548
I20260926 20:56:24.394424 8441 tablet_replica.cc:330] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35: stopping tablet replica
I20260926 20:56:24.394428 8442 tablet_replica.cc:330] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742: stopping tablet replica
I20260926 20:56:24.394430 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 20:56:24.394532 8441 raft_consensus.cc:2229] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 20:56:24.394573 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:56:24.394657 8441 raft_consensus.cc:2258] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:24.394785 8442 raft_consensus.cc:2229] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:56:24.394932 8442 raft_consensus.cc:2258] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:24.395133 8441 ts_tablet_manager.cc:1905] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:24.395717 8442 ts_tablet_manager.cc:1905] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:24.395890 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 8eaf8c4bb46843b9a7eb2fd67f24c1e7 (DEFAULT_TABLE table=test-table [id=8bb5726a8a0a45238422730b0011930b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:24.396024 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eaf8c4bb46843b9a7eb2fd67f24c1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:24.396577 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 8eaf8c4bb46843b9a7eb2fd67f24c1e7 (DEFAULT_TABLE table=test-table [id=8bb5726a8a0a45238422730b0011930b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:24.396687 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eaf8c4bb46843b9a7eb2fd67f24c1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:24.397105 8441 ts_tablet_manager.cc:1918] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:24.397324 8441 log.cc:1196] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/wal/wals/480166ab0e3f40c3a9d65d8c594110ec
I20260926 20:56:24.397605 8441 ts_tablet_manager.cc:1939] T 480166ab0e3f40c3a9d65d8c594110ec P 0552e62e93104b9488582a9b72bc0c35: Deleting consensus metadata
I20260926 20:56:24.397998 8442 ts_tablet_manager.cc:1918] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:24.398113 8442 log.cc:1196] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/480166ab0e3f40c3a9d65d8c594110ec
I20260926 20:56:24.398149 8418 tablet_bootstrap.cc:492] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35: Bootstrap starting.
I20260926 20:56:24.398275 7893 catalog_manager.cc:4841] TS 0552e62e93104b9488582a9b72bc0c35 (127.0.98.1:39217): tablet 480166ab0e3f40c3a9d65d8c594110ec (table test-table [id=8c6764d3f85c4305b07556c5f55a02ae]) successfully deleted
I20260926 20:56:24.398491 8442 ts_tablet_manager.cc:1939] T 480166ab0e3f40c3a9d65d8c594110ec P f79ca367d62b46bdb32256e3cfcf5742: Deleting consensus metadata
I20260926 20:56:24.398631 8418 tablet_bootstrap.cc:654] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:24.399238 8418 tablet_bootstrap.cc:492] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35: No bootstrap required, opened a new log
I20260926 20:56:24.399292 8418 ts_tablet_manager.cc:1397] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:24.399448 8418 raft_consensus.cc:348] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.399528 8418 raft_consensus.cc:374] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:24.399551 8418 raft_consensus.cc:729] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0552e62e93104b9488582a9b72bc0c35, State: Initialized, Role: FOLLOWER
I20260926 20:56:24.399603 8418 consensus_queue.cc:260] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.399605 8417 tablet_bootstrap.cc:492] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0: Bootstrap starting.
I20260926 20:56:24.399688 8418 ts_tablet_manager.cc:1428] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:24.399912 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 8eaf8c4bb46843b9a7eb2fd67f24c1e7 (DEFAULT_TABLE table=test-table [id=8bb5726a8a0a45238422730b0011930b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:24.400007 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eaf8c4bb46843b9a7eb2fd67f24c1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:24.400094 8417 tablet_bootstrap.cc:654] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:24.400779 8417 tablet_bootstrap.cc:492] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0: No bootstrap required, opened a new log
I20260926 20:56:24.400822 8417 ts_tablet_manager.cc:1397] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:24.400983 8417 raft_consensus.cc:348] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.401062 8417 raft_consensus.cc:374] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:24.401082 8417 raft_consensus.cc:729] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78209d3846f8467bb3fd656dffeb72c0, State: Initialized, Role: FOLLOWER
I20260926 20:56:24.401129 8417 consensus_queue.cc:260] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.401223 8417 ts_tablet_manager.cc:1428] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 20:56:24.401360 8419 tablet_bootstrap.cc:492] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742: Bootstrap starting.
I20260926 20:56:24.401399 7893 catalog_manager.cc:4841] TS f79ca367d62b46bdb32256e3cfcf5742 (127.0.98.2:35471): tablet 480166ab0e3f40c3a9d65d8c594110ec (table test-table [id=8c6764d3f85c4305b07556c5f55a02ae]) successfully deleted
I20260926 20:56:24.401782 8419 tablet_bootstrap.cc:654] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:24.402017 8440 tablet_replica.cc:330] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0: stopping tablet replica
I20260926 20:56:24.402137 8440 raft_consensus.cc:2229] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:24.402263 8440 raft_consensus.cc:2258] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:24.402427 8419 tablet_bootstrap.cc:492] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742: No bootstrap required, opened a new log
I20260926 20:56:24.402482 8419 ts_tablet_manager.cc:1397] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:24.402630 8419 raft_consensus.cc:348] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.402704 8419 raft_consensus.cc:374] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:24.402725 8419 raft_consensus.cc:729] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f79ca367d62b46bdb32256e3cfcf5742, State: Initialized, Role: FOLLOWER
I20260926 20:56:24.402735 8440 ts_tablet_manager.cc:1905] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:24.402769 8419 consensus_queue.cc:260] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.402850 8419 ts_tablet_manager.cc:1428] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:24.404278 8440 ts_tablet_manager.cc:1918] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:24.404366 8440 log.cc:1196] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/480166ab0e3f40c3a9d65d8c594110ec
I20260926 20:56:24.404666 8440 ts_tablet_manager.cc:1939] T 480166ab0e3f40c3a9d65d8c594110ec P 78209d3846f8467bb3fd656dffeb72c0: Deleting consensus metadata
I20260926 20:56:24.405087 7892 catalog_manager.cc:4841] TS 78209d3846f8467bb3fd656dffeb72c0 (127.0.98.3:38091): tablet 480166ab0e3f40c3a9d65d8c594110ec (table test-table [id=8c6764d3f85c4305b07556c5f55a02ae]) successfully deleted
W20260926 20:56:24.441308 8110 tablet.cc:2369] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:56:24.459404 8423 raft_consensus.cc:482] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:56:24.459636 8423 raft_consensus.cc:504] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.460083 8423 leader_election.cc:288] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f79ca367d62b46bdb32256e3cfcf5742 (127.0.98.2:35471), 78209d3846f8467bb3fd656dffeb72c0 (127.0.98.3:38091)
I20260926 20:56:24.463805 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.460271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38091 (local address 127.0.98.1:57809)
0926 20:56:24.460467 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:24.460478 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:24.460620 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:24.460791 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:24.460792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:24.460795 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:24.460875 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.460877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.461867 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.461869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.462797 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.462800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.463506 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.463507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.463590 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.463595 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:56:24.463660 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:56:24.463708 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":55,"thread_start_us":59,"threads_started":1}
I20260926 20:56:24.463804 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.460383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57809 (local address 127.0.98.3:38091)
0926 20:56:24.460457 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.460460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.460593 (+ 133us) server_negotiation.cc:407] Connection header received
0926 20:56:24.460716 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.460717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.460726 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.460789 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:24.460954 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.461736 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.462900 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.463417 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.463491 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.463503 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.463675 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.463725 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:56:24.464115 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.460386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47265 (local address 127.0.98.2:35471)
0926 20:56:24.460501 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.460503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.460605 (+ 102us) server_negotiation.cc:407] Connection header received
0926 20:56:24.460708 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.460710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.460721 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.460775 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:24.461020 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.461735 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.463180 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.463693 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.463755 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.463765 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.463959 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.464046 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 20:56:24.464382 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.460271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35471 (local address 127.0.98.1:47265)
0926 20:56:24.460545 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:24.460555 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:24.460612 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:24.460834 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:24.460835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:24.460838 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:24.460903 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.460905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.461837 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.461838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.463081 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.463085 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.463799 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.463800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.463872 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.463876 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:56:24.464245 (+ 369us) client_negotiation.cc:240] Negotiation successful
0926 20:56:24.464299 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":168,"thread_start_us":45,"threads_started":1}
I20260926 20:56:24.464488 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eaf8c4bb46843b9a7eb2fd67f24c1e7" candidate_uuid: "0552e62e93104b9488582a9b72bc0c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78209d3846f8467bb3fd656dffeb72c0" is_pre_election: true
I20260926 20:56:24.464596 8343 raft_consensus.cc:2454] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0552e62e93104b9488582a9b72bc0c35 in term 0.
I20260926 20:56:24.464684 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eaf8c4bb46843b9a7eb2fd67f24c1e7" candidate_uuid: "0552e62e93104b9488582a9b72bc0c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79ca367d62b46bdb32256e3cfcf5742" is_pre_election: true
I20260926 20:56:24.464789 7996 leader_election.cc:302] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0552e62e93104b9488582a9b72bc0c35, 78209d3846f8467bb3fd656dffeb72c0; no voters:
I20260926 20:56:24.464828 8203 raft_consensus.cc:2454] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0552e62e93104b9488582a9b72bc0c35 in term 0.
I20260926 20:56:24.464932 8423 raft_consensus.cc:2781] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:56:24.464970 8423 raft_consensus.cc:482] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:56:24.464996 8423 raft_consensus.cc:3037] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:24.465881 8423 raft_consensus.cc:504] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.466032 8423 leader_election.cc:288] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [CANDIDATE]: Term 1 election: Requested vote from peers f79ca367d62b46bdb32256e3cfcf5742 (127.0.98.2:35471), 78209d3846f8467bb3fd656dffeb72c0 (127.0.98.3:38091)
I20260926 20:56:24.466174 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eaf8c4bb46843b9a7eb2fd67f24c1e7" candidate_uuid: "0552e62e93104b9488582a9b72bc0c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79ca367d62b46bdb32256e3cfcf5742"
I20260926 20:56:24.466252 8203 raft_consensus.cc:3037] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:24.466259 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eaf8c4bb46843b9a7eb2fd67f24c1e7" candidate_uuid: "0552e62e93104b9488582a9b72bc0c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78209d3846f8467bb3fd656dffeb72c0"
I20260926 20:56:24.466347 8343 raft_consensus.cc:3037] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:24.466902 8203 raft_consensus.cc:2454] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0552e62e93104b9488582a9b72bc0c35 in term 1.
I20260926 20:56:24.467098 7997 leader_election.cc:302] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0552e62e93104b9488582a9b72bc0c35, f79ca367d62b46bdb32256e3cfcf5742; no voters:
I20260926 20:56:24.467130 8343 raft_consensus.cc:2454] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0552e62e93104b9488582a9b72bc0c35 in term 1.
I20260926 20:56:24.467195 8423 raft_consensus.cc:2781] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:24.467308 8423 raft_consensus.cc:686] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [term 1 LEADER]: Becoming Leader. State: Replica: 0552e62e93104b9488582a9b72bc0c35, State: Running, Role: LEADER
I20260926 20:56:24.467379 8423 consensus_queue.cc:237] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.468178 7904 catalog_manager.cc:5495] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0552e62e93104b9488582a9b72bc0c35 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0552e62e93104b9488582a9b72bc0c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } health_report { overall_health: UNKNOWN } } }
Sep 26 20:56:24 dist-test-slave-fl7g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456183, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:56:24.501385 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.493463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60482 (local address 127.0.98.62:45723)
0926 20:56:24.493534 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.493536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.493661 (+ 125us) server_negotiation.cc:407] Connection header received
0926 20:56:24.493724 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.493726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.493738 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.493822 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:24.494366 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.495138 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.495886 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.496205 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.496288 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.498623 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:24.498624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:24.498625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:24.498632 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:24.500057 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:24.500535 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:24.500536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:24.500536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:24.500585 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:24.500794 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:24.500795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:24.500795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:24.500898 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:24.500949 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.501150 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.501291 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":39}
I20260926 20:56:24.501386 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.493366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45723 (local address 127.0.0.1:60482)
0926 20:56:24.493572 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:24.493586 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:24.493667 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:24.493840 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:24.493841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:24.494204 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:24.494264 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.494266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.495242 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.495243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.495789 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.495791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.496293 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.496294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.496336 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.497014 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:24.497019 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:24.498530 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:24.500148 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:24.500157 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:24.500159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:24.500458 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:24.500651 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:24.500651 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:24.500652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:24.500731 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:24.500963 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:24.500971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:24.501028 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 20:56:24.501114 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 20:56:24.501246 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":138,"threads_started":1}
I20260926 20:56:24.503666 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60482:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:56:24.503924 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:56:24.505343 8323 tablet_service.cc:1467] Processing CreateTablet for tablet f360e4edb9e347ec9c92966cb55e34f3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=49ab449cbe1d47eda4ce9bfdfb85572f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:56:24.505455 8183 tablet_service.cc:1467] Processing CreateTablet for tablet f360e4edb9e347ec9c92966cb55e34f3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=49ab449cbe1d47eda4ce9bfdfb85572f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:56:24.505471 8043 tablet_service.cc:1467] Processing CreateTablet for tablet f360e4edb9e347ec9c92966cb55e34f3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=49ab449cbe1d47eda4ce9bfdfb85572f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:56:24.505537 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f360e4edb9e347ec9c92966cb55e34f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:24.505568 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f360e4edb9e347ec9c92966cb55e34f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:24.505625 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f360e4edb9e347ec9c92966cb55e34f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:24.507107 8417 tablet_bootstrap.cc:492] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0: Bootstrap starting.
I20260926 20:56:24.507107 8419 tablet_bootstrap.cc:492] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742: Bootstrap starting.
I20260926 20:56:24.507220 8418 tablet_bootstrap.cc:492] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35: Bootstrap starting.
I20260926 20:56:24.507562 8419 tablet_bootstrap.cc:654] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:24.507562 8417 tablet_bootstrap.cc:654] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:24.507652 8418 tablet_bootstrap.cc:654] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:24.508567 8419 tablet_bootstrap.cc:492] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742: No bootstrap required, opened a new log
I20260926 20:56:24.508621 8419 ts_tablet_manager.cc:1397] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:56:24.508773 8419 raft_consensus.cc:348] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.508838 8419 raft_consensus.cc:374] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:24.508855 8419 raft_consensus.cc:729] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f79ca367d62b46bdb32256e3cfcf5742, State: Initialized, Role: FOLLOWER
I20260926 20:56:24.508895 8419 consensus_queue.cc:260] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.508948 8417 tablet_bootstrap.cc:492] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0: No bootstrap required, opened a new log
I20260926 20:56:24.508972 8419 ts_tablet_manager.cc:1428] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:24.508993 8417 ts_tablet_manager.cc:1397] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:56:24.509088 8427 tablet_replica.cc:439] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } }
I20260926 20:56:24.509145 8427 tablet_replica.cc:442] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:56:24.509138 8417 raft_consensus.cc:348] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.509214 8417 raft_consensus.cc:374] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:24.509240 8417 raft_consensus.cc:729] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78209d3846f8467bb3fd656dffeb72c0, State: Initialized, Role: FOLLOWER
I20260926 20:56:24.509284 8417 consensus_queue.cc:260] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.509361 8417 ts_tablet_manager.cc:1428] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 20:56:24.509405 8418 tablet_bootstrap.cc:492] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35: No bootstrap required, opened a new log
I20260926 20:56:24.509444 8418 ts_tablet_manager.cc:1397] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:56:24.509518 8424 tablet_replica.cc:439] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } }
I20260926 20:56:24.509579 8418 raft_consensus.cc:348] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.509635 8418 raft_consensus.cc:374] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:24.509613 8424 tablet_replica.cc:442] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:56:24.509653 8418 raft_consensus.cc:729] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0552e62e93104b9488582a9b72bc0c35, State: Initialized, Role: FOLLOWER
I20260926 20:56:24.509691 8418 consensus_queue.cc:260] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.509763 8418 ts_tablet_manager.cc:1428] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:24.509879 8448 tablet_replica.cc:439] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } }
I20260926 20:56:24.509943 8448 tablet_replica.cc:442] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 20:56:24.519469 8390 tablet.cc:2369] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:56:24.619659 8448 raft_consensus.cc:482] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:56:24.619771 8448 raft_consensus.cc:504] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.619946 8448 leader_election.cc:288] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f79ca367d62b46bdb32256e3cfcf5742 (127.0.98.2:35471), 78209d3846f8467bb3fd656dffeb72c0 (127.0.98.3:38091)
I20260926 20:56:24.620261 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f360e4edb9e347ec9c92966cb55e34f3" candidate_uuid: "0552e62e93104b9488582a9b72bc0c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78209d3846f8467bb3fd656dffeb72c0" is_pre_election: true
I20260926 20:56:24.620258 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f360e4edb9e347ec9c92966cb55e34f3" candidate_uuid: "0552e62e93104b9488582a9b72bc0c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79ca367d62b46bdb32256e3cfcf5742" is_pre_election: true
I20260926 20:56:24.620352 8343 raft_consensus.cc:2454] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0552e62e93104b9488582a9b72bc0c35 in term 0.
I20260926 20:56:24.620378 8203 raft_consensus.cc:2454] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0552e62e93104b9488582a9b72bc0c35 in term 0.
I20260926 20:56:24.620560 7997 leader_election.cc:302] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0552e62e93104b9488582a9b72bc0c35, f79ca367d62b46bdb32256e3cfcf5742; no voters:
I20260926 20:56:24.620703 8448 raft_consensus.cc:2781] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:56:24.620757 8448 raft_consensus.cc:482] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:56:24.620781 8448 raft_consensus.cc:3037] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:24.621543 8448 raft_consensus.cc:504] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.621690 8448 leader_election.cc:288] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [CANDIDATE]: Term 1 election: Requested vote from peers f79ca367d62b46bdb32256e3cfcf5742 (127.0.98.2:35471), 78209d3846f8467bb3fd656dffeb72c0 (127.0.98.3:38091)
I20260926 20:56:24.621840 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f360e4edb9e347ec9c92966cb55e34f3" candidate_uuid: "0552e62e93104b9488582a9b72bc0c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79ca367d62b46bdb32256e3cfcf5742"
I20260926 20:56:24.621932 8203 raft_consensus.cc:3037] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:24.621883 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f360e4edb9e347ec9c92966cb55e34f3" candidate_uuid: "0552e62e93104b9488582a9b72bc0c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78209d3846f8467bb3fd656dffeb72c0"
I20260926 20:56:24.621992 8343 raft_consensus.cc:3037] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:24.622602 8203 raft_consensus.cc:2454] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0552e62e93104b9488582a9b72bc0c35 in term 1.
I20260926 20:56:24.622613 8343 raft_consensus.cc:2454] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0552e62e93104b9488582a9b72bc0c35 in term 1.
I20260926 20:56:24.622822 7997 leader_election.cc:302] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0552e62e93104b9488582a9b72bc0c35, f79ca367d62b46bdb32256e3cfcf5742; no voters:
I20260926 20:56:24.622910 8448 raft_consensus.cc:2781] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:24.622977 8448 raft_consensus.cc:686] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [term 1 LEADER]: Becoming Leader. State: Replica: 0552e62e93104b9488582a9b72bc0c35, State: Running, Role: LEADER
I20260926 20:56:24.623031 8448 consensus_queue.cc:237] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.623399 8423 tablet_replica.cc:439] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35: TxnStatusTablet state changed. Reason: New leader 0552e62e93104b9488582a9b72bc0c35. Latest consensus state: current_term: 1 leader_uuid: "0552e62e93104b9488582a9b72bc0c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } }
I20260926 20:56:24.623502 8423 tablet_replica.cc:442] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:56:24.623703 8460 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:56:24.623879 8460 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:56:24.623884 7904 catalog_manager.cc:5495] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0552e62e93104b9488582a9b72bc0c35 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0552e62e93104b9488582a9b72bc0c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } health_report { overall_health: UNKNOWN } } }
I20260926 20:56:24.657919 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.655296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39217 (local address 127.0.0.1:35728)
0926 20:56:24.655386 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:24.655397 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:24.655463 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:24.655649 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:24.655650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:24.655652 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:56:24.655704 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.655712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.656364 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.656365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.656971 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.656973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.657368 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.657369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.657437 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.657451 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:56:24.657746 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:56:24.657755 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 20:56:24.657800 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 20:56:24.657846 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 20:56:24.657948 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.655389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35728 (local address 127.0.98.1:39217)
0926 20:56:24.655455 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.655457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.655467 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:56:24.655553 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.655554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.655576 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.655635 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:24.655778 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.656280 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.657063 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.657270 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.657362 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.657565 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:24.657669 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:24.657744 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.657802 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.657824 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:56:24.659164 8343 raft_consensus.cc:1261] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0 [term 1 FOLLOWER]: Refusing update from remote peer 0552e62e93104b9488582a9b72bc0c35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:24.659176 8203 raft_consensus.cc:1261] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742 [term 1 FOLLOWER]: Refusing update from remote peer 0552e62e93104b9488582a9b72bc0c35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:24.659540 8423 consensus_queue.cc:1035] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:24.659693 8448 consensus_queue.cc:1035] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:24.660231 8424 tablet_replica.cc:439] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0: TxnStatusTablet state changed. Reason: New leader 0552e62e93104b9488582a9b72bc0c35. Latest consensus state: current_term: 1 leader_uuid: "0552e62e93104b9488582a9b72bc0c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } }
I20260926 20:56:24.660256 8427 tablet_replica.cc:439] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742: TxnStatusTablet state changed. Reason: New leader 0552e62e93104b9488582a9b72bc0c35. Latest consensus state: current_term: 1 leader_uuid: "0552e62e93104b9488582a9b72bc0c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } }
I20260926 20:56:24.660327 8424 tablet_replica.cc:442] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:56:24.660332 8427 tablet_replica.cc:442] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:56:24.661166 8448 tablet_replica.cc:439] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0552e62e93104b9488582a9b72bc0c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } }
I20260926 20:56:24.661247 8448 tablet_replica.cc:442] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:56:24.661321 8424 tablet_replica.cc:439] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0552e62e93104b9488582a9b72bc0c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } }
I20260926 20:56:24.661414 8424 tablet_replica.cc:442] T f360e4edb9e347ec9c92966cb55e34f3 P 78209d3846f8467bb3fd656dffeb72c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:56:24.661901 8427 tablet_replica.cc:439] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0552e62e93104b9488582a9b72bc0c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } }
I20260926 20:56:24.661988 8427 tablet_replica.cc:442] T f360e4edb9e347ec9c92966cb55e34f3 P f79ca367d62b46bdb32256e3cfcf5742: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:56:24.662088 8423 tablet_replica.cc:439] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0552e62e93104b9488582a9b72bc0c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } } }
I20260926 20:56:24.662168 8423 tablet_replica.cc:442] T f360e4edb9e347ec9c92966cb55e34f3 P 0552e62e93104b9488582a9b72bc0c35: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:56:24.665738 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.663106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35730 (local address 127.0.98.1:39217)
0926 20:56:24.663146 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.663148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.663162 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:56:24.663235 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.663236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.663246 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.663315 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:24.663438 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.664200 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.665046 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.665348 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.665422 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.665518 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:24.665558 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:24.665610 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.665646 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.665664 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:56:24.671015 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.668348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34607 (local address 127.0.98.1:39217)
0926 20:56:24.668387 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.668389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.668537 (+ 148us) server_negotiation.cc:407] Connection header received
0926 20:56:24.668593 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.668594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.668604 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.668668 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:24.668785 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.669471 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.670253 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.670542 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.670619 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.670788 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:24.670818 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:24.670866 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.670915 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.670932 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:56:24.671028 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.668293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39217 (local address 127.0.98.1:34607)
0926 20:56:24.668464 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:24.668475 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:24.668536 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:24.668671 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:24.668671 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:24.668672 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:56:24.668735 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.668736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.669559 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.669560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.670147 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:24.670148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.670644 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.670645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:24.670697 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.670711 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:56:24.670873 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:56:24.670874 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:56:24.670905 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 20:56:24.670942 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260926 20:56:24.673450 8203 raft_consensus.cc:1261] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742 [term 1 FOLLOWER]: Refusing update from remote peer 0552e62e93104b9488582a9b72bc0c35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:24.673449 8343 raft_consensus.cc:1261] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0 [term 1 FOLLOWER]: Refusing update from remote peer 0552e62e93104b9488582a9b72bc0c35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:24.673663 8465 consensus_queue.cc:1035] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:24.673784 8448 consensus_queue.cc:1035] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:24.685477 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:24.685635 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:24.686230 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca: Sending DeleteTablet for 3 replicas of tablet 8eaf8c4bb46843b9a7eb2fd67f24c1e7
I20260926 20:56:24.686559 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eaf8c4bb46843b9a7eb2fd67f24c1e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43410
I20260926 20:56:24.686564 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eaf8c4bb46843b9a7eb2fd67f24c1e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35714
I20260926 20:56:24.686693 8441 tablet_replica.cc:330] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35: stopping tablet replica
I20260926 20:56:24.686754 8441 raft_consensus.cc:2229] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:56:24.686846 8441 raft_consensus.cc:2258] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:24.686930 8440 tablet_replica.cc:330] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0: stopping tablet replica
I20260926 20:56:24.686993 8440 raft_consensus.cc:2229] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:24.687038 8440 raft_consensus.cc:2258] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:24.687171 8441 ts_tablet_manager.cc:1905] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:24.687180 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eaf8c4bb46843b9a7eb2fd67f24c1e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51548
I20260926 20:56:24.687281 8442 tablet_replica.cc:330] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742: stopping tablet replica
I20260926 20:56:24.687358 8442 raft_consensus.cc:2229] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:24.687368 8440 ts_tablet_manager.cc:1905] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:24.687400 8442 raft_consensus.cc:2258] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:24.688886 8441 ts_tablet_manager.cc:1918] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:56:24.688947 8441 log.cc:1196] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/wal/wals/8eaf8c4bb46843b9a7eb2fd67f24c1e7
I20260926 20:56:24.689162 8441 ts_tablet_manager.cc:1939] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 0552e62e93104b9488582a9b72bc0c35: Deleting consensus metadata
I20260926 20:56:24.689285 8440 ts_tablet_manager.cc:1918] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:56:24.689352 8440 log.cc:1196] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/8eaf8c4bb46843b9a7eb2fd67f24c1e7
I20260926 20:56:24.689575 8440 ts_tablet_manager.cc:1939] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P 78209d3846f8467bb3fd656dffeb72c0: Deleting consensus metadata
I20260926 20:56:24.689937 7892 catalog_manager.cc:4841] TS 78209d3846f8467bb3fd656dffeb72c0 (127.0.98.3:38091): tablet 8eaf8c4bb46843b9a7eb2fd67f24c1e7 (table test-table [id=8bb5726a8a0a45238422730b0011930b]) successfully deleted
I20260926 20:56:24.689992 7893 catalog_manager.cc:4841] TS 0552e62e93104b9488582a9b72bc0c35 (127.0.98.1:39217): tablet 8eaf8c4bb46843b9a7eb2fd67f24c1e7 (table test-table [id=8bb5726a8a0a45238422730b0011930b]) successfully deleted
I20260926 20:56:24.690868 8442 ts_tablet_manager.cc:1905] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:24.692471 8442 ts_tablet_manager.cc:1918] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:56:24.692529 8442 log.cc:1196] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/8eaf8c4bb46843b9a7eb2fd67f24c1e7
I20260926 20:56:24.692731 8442 ts_tablet_manager.cc:1939] T 8eaf8c4bb46843b9a7eb2fd67f24c1e7 P f79ca367d62b46bdb32256e3cfcf5742: Deleting consensus metadata
I20260926 20:56:24.693192 7893 catalog_manager.cc:4841] TS f79ca367d62b46bdb32256e3cfcf5742 (127.0.98.2:35471): tablet 8eaf8c4bb46843b9a7eb2fd67f24c1e7 (table test-table [id=8bb5726a8a0a45238422730b0011930b]) successfully deleted
Sep 26 20:56:24 dist-test-slave-fl7g krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:56:24 dist-test-slave-fl7g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:24 dist-test-slave-fl7g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:56:24.712636 7956 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:24.705729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60496 (local address 127.0.98.62:45723)
0926 20:56:24.705784 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.705786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.705901 (+ 115us) server_negotiation.cc:407] Connection header received
0926 20:56:24.705983 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.705984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.705993 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.706046 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:24.706818 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.707308 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.708535 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.708775 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.708858 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.712549 (+ 3691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60496: BlockingRecv error: recv got EOF from 127.0.0.1:60496 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 26 20:56:24 dist-test-slave-fl7g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:56:24.721850 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.713568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60500 (local address 127.0.98.62:45723)
0926 20:56:24.713612 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.713614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.713681 (+ 67us) server_negotiation.cc:407] Connection header received
0926 20:56:24.713749 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.713750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.713764 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.713813 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:24.714409 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.714957 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.716027 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.716243 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.716309 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.718995 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:24.718996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:24.718996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:24.719006 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:24.720607 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:24.721040 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:24.721041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:24.721042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:24.721100 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:24.721287 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:24.721288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:24.721289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:24.721402 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:24.721498 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.721661 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.721744 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:56:24.723501 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:56:24.723691 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:56:24.725129 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 287597caf54b454f935cbe8e5c0e66aa (DEFAULT_TABLE table=test-table [id=4c92189e574a4da2ba5cbd22a2fc1196]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:24.725133 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 287597caf54b454f935cbe8e5c0e66aa (DEFAULT_TABLE table=test-table [id=4c92189e574a4da2ba5cbd22a2fc1196]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:24.725273 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 287597caf54b454f935cbe8e5c0e66aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:24.725279 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 287597caf54b454f935cbe8e5c0e66aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:24.726212 8419 tablet_bootstrap.cc:492] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742: Bootstrap starting.
I20260926 20:56:24.726604 8419 tablet_bootstrap.cc:654] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:24.726663 8418 tablet_bootstrap.cc:492] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35: Bootstrap starting.
I20260926 20:56:24.726722 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 287597caf54b454f935cbe8e5c0e66aa (DEFAULT_TABLE table=test-table [id=4c92189e574a4da2ba5cbd22a2fc1196]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:24.726841 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 287597caf54b454f935cbe8e5c0e66aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:24.727098 8418 tablet_bootstrap.cc:654] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:24.727723 8418 tablet_bootstrap.cc:492] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35: No bootstrap required, opened a new log
I20260926 20:56:24.727756 8418 ts_tablet_manager.cc:1397] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:24.727881 8418 raft_consensus.cc:348] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.727939 8418 raft_consensus.cc:374] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:24.727953 8418 raft_consensus.cc:729] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0552e62e93104b9488582a9b72bc0c35, State: Initialized, Role: FOLLOWER
I20260926 20:56:24.728010 8418 consensus_queue.cc:260] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.728106 8418 ts_tablet_manager.cc:1428] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:24.728276 8417 tablet_bootstrap.cc:492] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0: Bootstrap starting.
I20260926 20:56:24.729058 8417 tablet_bootstrap.cc:654] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:24.729131 8419 tablet_bootstrap.cc:492] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742: No bootstrap required, opened a new log
I20260926 20:56:24.729188 8419 ts_tablet_manager.cc:1397] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 20:56:24.729329 8419 raft_consensus.cc:348] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.729401 8419 raft_consensus.cc:374] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:24.729423 8419 raft_consensus.cc:729] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f79ca367d62b46bdb32256e3cfcf5742, State: Initialized, Role: FOLLOWER
I20260926 20:56:24.729476 8419 consensus_queue.cc:260] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.729576 8419 ts_tablet_manager.cc:1428] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:24.729710 8417 tablet_bootstrap.cc:492] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0: No bootstrap required, opened a new log
I20260926 20:56:24.729743 8417 ts_tablet_manager.cc:1397] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:24.729884 8417 raft_consensus.cc:348] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.729969 8417 raft_consensus.cc:374] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:24.730005 8417 raft_consensus.cc:729] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78209d3846f8467bb3fd656dffeb72c0, State: Initialized, Role: FOLLOWER
I20260926 20:56:24.730044 8417 consensus_queue.cc:260] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.730165 8417 ts_tablet_manager.cc:1428] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:24.750741 8427 raft_consensus.cc:482] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:56:24.750851 8427 raft_consensus.cc:504] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.751042 8427 leader_election.cc:288] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0552e62e93104b9488582a9b72bc0c35 (127.0.98.1:39217), 78209d3846f8467bb3fd656dffeb72c0 (127.0.98.3:38091)
I20260926 20:56:24.751235 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287597caf54b454f935cbe8e5c0e66aa" candidate_uuid: "f79ca367d62b46bdb32256e3cfcf5742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0552e62e93104b9488582a9b72bc0c35" is_pre_election: true
I20260926 20:56:24.751272 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287597caf54b454f935cbe8e5c0e66aa" candidate_uuid: "f79ca367d62b46bdb32256e3cfcf5742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78209d3846f8467bb3fd656dffeb72c0" is_pre_election: true
I20260926 20:56:24.751338 8063 raft_consensus.cc:2454] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f79ca367d62b46bdb32256e3cfcf5742 in term 0.
I20260926 20:56:24.751350 8343 raft_consensus.cc:2454] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f79ca367d62b46bdb32256e3cfcf5742 in term 0.
I20260926 20:56:24.751492 8136 leader_election.cc:302] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78209d3846f8467bb3fd656dffeb72c0, f79ca367d62b46bdb32256e3cfcf5742; no voters:
I20260926 20:56:24.751616 8427 raft_consensus.cc:2781] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:56:24.751665 8427 raft_consensus.cc:482] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:56:24.751698 8427 raft_consensus.cc:3037] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:24.752533 8427 raft_consensus.cc:504] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.752691 8427 leader_election.cc:288] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [CANDIDATE]: Term 1 election: Requested vote from peers 0552e62e93104b9488582a9b72bc0c35 (127.0.98.1:39217), 78209d3846f8467bb3fd656dffeb72c0 (127.0.98.3:38091)
I20260926 20:56:24.752830 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287597caf54b454f935cbe8e5c0e66aa" candidate_uuid: "f79ca367d62b46bdb32256e3cfcf5742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78209d3846f8467bb3fd656dffeb72c0"
I20260926 20:56:24.752924 8343 raft_consensus.cc:3037] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:24.752951 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287597caf54b454f935cbe8e5c0e66aa" candidate_uuid: "f79ca367d62b46bdb32256e3cfcf5742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0552e62e93104b9488582a9b72bc0c35"
I20260926 20:56:24.753031 8063 raft_consensus.cc:3037] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:24.753762 8343 raft_consensus.cc:2454] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f79ca367d62b46bdb32256e3cfcf5742 in term 1.
I20260926 20:56:24.753855 8063 raft_consensus.cc:2454] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f79ca367d62b46bdb32256e3cfcf5742 in term 1.
I20260926 20:56:24.754029 8136 leader_election.cc:302] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78209d3846f8467bb3fd656dffeb72c0, f79ca367d62b46bdb32256e3cfcf5742; no voters:
I20260926 20:56:24.754146 8427 raft_consensus.cc:2781] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:24.754199 8427 raft_consensus.cc:686] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [term 1 LEADER]: Becoming Leader. State: Replica: f79ca367d62b46bdb32256e3cfcf5742, State: Running, Role: LEADER
I20260926 20:56:24.754245 8427 consensus_queue.cc:237] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.754863 7906 catalog_manager.cc:5495] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 reported cstate change: term changed from 0 to 1, leader changed from <none> to f79ca367d62b46bdb32256e3cfcf5742 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f79ca367d62b46bdb32256e3cfcf5742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } health_report { overall_health: UNKNOWN } } }
I20260926 20:56:24.766319 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:24.763974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.0.98.2:35471)
0926 20:56:24.764027 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:24.764029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:24.764039 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:56:24.764104 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:24.764105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:24.764119 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:24.764189 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:24.764337 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.764865 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.765620 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:24.765816 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:24.765886 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:24.766044 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:24.766082 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:24.766141 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:24.766204 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 20:56:24.766226 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 20:56:24.767112 8343 raft_consensus.cc:1261] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0 [term 1 FOLLOWER]: Refusing update from remote peer f79ca367d62b46bdb32256e3cfcf5742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:24.767138 8063 raft_consensus.cc:1261] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35 [term 1 FOLLOWER]: Refusing update from remote peer f79ca367d62b46bdb32256e3cfcf5742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:24.767333 8427 consensus_queue.cc:1035] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:24.767423 8437 consensus_queue.cc:1035] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:24.770349 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:24.770443 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:24.771111 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca: Sending DeleteTablet for 3 replicas of tablet 287597caf54b454f935cbe8e5c0e66aa
I20260926 20:56:24.771373 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 287597caf54b454f935cbe8e5c0e66aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43410
I20260926 20:56:24.771463 8440 tablet_replica.cc:330] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0: stopping tablet replica
I20260926 20:56:24.771507 8440 raft_consensus.cc:2229] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:24.771544 8440 raft_consensus.cc:2258] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:24.771869 8440 ts_tablet_manager.cc:1905] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:24.771862 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:56:24.771998 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:56:24.772183 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 287597caf54b454f935cbe8e5c0e66aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35714
I20260926 20:56:24.772285 8441 tablet_replica.cc:330] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35: stopping tablet replica
I20260926 20:56:24.772332 8441 raft_consensus.cc:2229] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:24.772374 8441 raft_consensus.cc:2258] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:24.772675 8441 ts_tablet_manager.cc:1905] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:24.773685 8440 ts_tablet_manager.cc:1918] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:24.773757 8440 log.cc:1196] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/287597caf54b454f935cbe8e5c0e66aa
I20260926 20:56:24.773969 8440 ts_tablet_manager.cc:1939] T 287597caf54b454f935cbe8e5c0e66aa P 78209d3846f8467bb3fd656dffeb72c0: Deleting consensus metadata
I20260926 20:56:24.774117 8441 ts_tablet_manager.cc:1918] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:24.774171 8441 log.cc:1196] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-0/wal/wals/287597caf54b454f935cbe8e5c0e66aa
I20260926 20:56:24.774318 7892 catalog_manager.cc:4841] TS 78209d3846f8467bb3fd656dffeb72c0 (127.0.98.3:38091): tablet 287597caf54b454f935cbe8e5c0e66aa (table test-table [id=4c92189e574a4da2ba5cbd22a2fc1196]) successfully deleted
I20260926 20:56:24.774358 8441 ts_tablet_manager.cc:1939] T 287597caf54b454f935cbe8e5c0e66aa P 0552e62e93104b9488582a9b72bc0c35: Deleting consensus metadata
I20260926 20:56:24.774410 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet 287597caf54b454f935cbe8e5c0e66aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51548
I20260926 20:56:24.774511 8442 tablet_replica.cc:330] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742: stopping tablet replica
I20260926 20:56:24.774576 8442 raft_consensus.cc:2229] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:56:24.774641 8442 raft_consensus.cc:2258] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:24.774883 8442 ts_tablet_manager.cc:1905] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:24.775512 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 3c855ea38a0e4642851b925d4479371f (DEFAULT_TABLE table=test-table [id=29c5f3d65e6746608f6059464d037fda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:24.775632 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c855ea38a0e4642851b925d4479371f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:24.776098 8442 ts_tablet_manager.cc:1918] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:24.776137 8442 log.cc:1196] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/287597caf54b454f935cbe8e5c0e66aa
I20260926 20:56:24.776300 8442 ts_tablet_manager.cc:1939] T 287597caf54b454f935cbe8e5c0e66aa P f79ca367d62b46bdb32256e3cfcf5742: Deleting consensus metadata
I20260926 20:56:24.776780 8417 tablet_bootstrap.cc:492] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0: Bootstrap starting.
I20260926 20:56:24.776827 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 3c855ea38a0e4642851b925d4479371f (DEFAULT_TABLE table=test-table [id=29c5f3d65e6746608f6059464d037fda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:24.776922 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c855ea38a0e4642851b925d4479371f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:24.777135 8417 tablet_bootstrap.cc:654] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:24.777490 7893 catalog_manager.cc:4841] TS 0552e62e93104b9488582a9b72bc0c35 (127.0.98.1:39217): tablet 287597caf54b454f935cbe8e5c0e66aa (table test-table [id=4c92189e574a4da2ba5cbd22a2fc1196]) successfully deleted
I20260926 20:56:24.777731 8417 tablet_bootstrap.cc:492] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0: No bootstrap required, opened a new log
I20260926 20:56:24.777791 8417 ts_tablet_manager.cc:1397] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:24.777935 8417 raft_consensus.cc:348] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.777985 8417 raft_consensus.cc:374] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:24.777998 8417 raft_consensus.cc:729] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78209d3846f8467bb3fd656dffeb72c0, State: Initialized, Role: FOLLOWER
I20260926 20:56:24.778050 8417 consensus_queue.cc:260] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.778158 8417 ts_tablet_manager.cc:1428] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:24.778414 8418 tablet_bootstrap.cc:492] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35: Bootstrap starting.
I20260926 20:56:24.778796 8418 tablet_bootstrap.cc:654] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:24.779209 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 3c855ea38a0e4642851b925d4479371f (DEFAULT_TABLE table=test-table [id=29c5f3d65e6746608f6059464d037fda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:24.779317 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c855ea38a0e4642851b925d4479371f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:24.779516 7893 catalog_manager.cc:4841] TS f79ca367d62b46bdb32256e3cfcf5742 (127.0.98.2:35471): tablet 287597caf54b454f935cbe8e5c0e66aa (table test-table [id=4c92189e574a4da2ba5cbd22a2fc1196]) successfully deleted
I20260926 20:56:24.779726 8418 tablet_bootstrap.cc:492] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35: No bootstrap required, opened a new log
I20260926 20:56:24.779760 8418 ts_tablet_manager.cc:1397] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:24.779873 8418 raft_consensus.cc:348] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.779925 8418 raft_consensus.cc:374] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:24.779942 8418 raft_consensus.cc:729] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0552e62e93104b9488582a9b72bc0c35, State: Initialized, Role: FOLLOWER
I20260926 20:56:24.779978 8418 consensus_queue.cc:260] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.780061 8418 ts_tablet_manager.cc:1428] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:24.780634 8419 tablet_bootstrap.cc:492] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742: Bootstrap starting.
I20260926 20:56:24.780951 8419 tablet_bootstrap.cc:654] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:24.781533 8419 tablet_bootstrap.cc:492] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742: No bootstrap required, opened a new log
I20260926 20:56:24.781629 8419 ts_tablet_manager.cc:1397] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:24.781766 8419 raft_consensus.cc:348] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.781890 8419 raft_consensus.cc:374] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:24.781917 8419 raft_consensus.cc:729] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f79ca367d62b46bdb32256e3cfcf5742, State: Initialized, Role: FOLLOWER
I20260926 20:56:24.781999 8419 consensus_queue.cc:260] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.782124 8419 ts_tablet_manager.cc:1428] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 20:56:24.960089 8471 raft_consensus.cc:482] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:56:24.960245 8471 raft_consensus.cc:504] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.960464 8471 leader_election.cc:288] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f79ca367d62b46bdb32256e3cfcf5742 (127.0.98.2:35471), 78209d3846f8467bb3fd656dffeb72c0 (127.0.98.3:38091)
I20260926 20:56:24.960706 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c855ea38a0e4642851b925d4479371f" candidate_uuid: "0552e62e93104b9488582a9b72bc0c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79ca367d62b46bdb32256e3cfcf5742" is_pre_election: true
I20260926 20:56:24.960791 8202 raft_consensus.cc:2454] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0552e62e93104b9488582a9b72bc0c35 in term 0.
I20260926 20:56:24.960903 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c855ea38a0e4642851b925d4479371f" candidate_uuid: "0552e62e93104b9488582a9b72bc0c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78209d3846f8467bb3fd656dffeb72c0" is_pre_election: true
I20260926 20:56:24.960953 7997 leader_election.cc:302] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0552e62e93104b9488582a9b72bc0c35, f79ca367d62b46bdb32256e3cfcf5742; no voters:
I20260926 20:56:24.960987 8343 raft_consensus.cc:2454] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0552e62e93104b9488582a9b72bc0c35 in term 0.
I20260926 20:56:24.961141 8471 raft_consensus.cc:2781] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:56:24.961189 8471 raft_consensus.cc:482] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:56:24.961208 8471 raft_consensus.cc:3037] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:24.961930 8471 raft_consensus.cc:504] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.962093 8471 leader_election.cc:288] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [CANDIDATE]: Term 1 election: Requested vote from peers f79ca367d62b46bdb32256e3cfcf5742 (127.0.98.2:35471), 78209d3846f8467bb3fd656dffeb72c0 (127.0.98.3:38091)
I20260926 20:56:24.962226 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c855ea38a0e4642851b925d4479371f" candidate_uuid: "0552e62e93104b9488582a9b72bc0c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79ca367d62b46bdb32256e3cfcf5742"
I20260926 20:56:24.962316 8202 raft_consensus.cc:3037] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:24.962272 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c855ea38a0e4642851b925d4479371f" candidate_uuid: "0552e62e93104b9488582a9b72bc0c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78209d3846f8467bb3fd656dffeb72c0"
I20260926 20:56:24.962365 8343 raft_consensus.cc:3037] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:24.962975 8343 raft_consensus.cc:2454] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0552e62e93104b9488582a9b72bc0c35 in term 1.
I20260926 20:56:24.963119 8202 raft_consensus.cc:2454] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0552e62e93104b9488582a9b72bc0c35 in term 1.
I20260926 20:56:24.963151 7996 leader_election.cc:302] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0552e62e93104b9488582a9b72bc0c35, 78209d3846f8467bb3fd656dffeb72c0; no voters:
I20260926 20:56:24.963234 8471 raft_consensus.cc:2781] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:24.963280 8471 raft_consensus.cc:686] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [term 1 LEADER]: Becoming Leader. State: Replica: 0552e62e93104b9488582a9b72bc0c35, State: Running, Role: LEADER
I20260926 20:56:24.963325 8471 consensus_queue.cc:237] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } }
I20260926 20:56:24.963907 7906 catalog_manager.cc:5495] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0552e62e93104b9488582a9b72bc0c35 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0552e62e93104b9488582a9b72bc0c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0552e62e93104b9488582a9b72bc0c35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 } health_report { overall_health: UNKNOWN } } }
I20260926 20:56:25.014459 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:25.012317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35738 (local address 127.0.98.1:39217)
0926 20:56:25.012356 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:25.012358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:25.012368 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:56:25.012484 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:25.012485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:25.012495 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:25.012565 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:25.012676 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:25.013184 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:25.013942 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:25.014126 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:25.014173 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:25.014275 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:25.014299 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:25.014330 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:25.014390 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 20:56:25.014402 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:56:25.016479 8343 raft_consensus.cc:1261] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0 [term 1 FOLLOWER]: Refusing update from remote peer 0552e62e93104b9488582a9b72bc0c35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:25.016587 8202 raft_consensus.cc:1261] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742 [term 1 FOLLOWER]: Refusing update from remote peer 0552e62e93104b9488582a9b72bc0c35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:25.016678 8465 consensus_queue.cc:1035] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78209d3846f8467bb3fd656dffeb72c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:25.016928 8465 consensus_queue.cc:1035] T 3c855ea38a0e4642851b925d4479371f P 0552e62e93104b9488582a9b72bc0c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79ca367d62b46bdb32256e3cfcf5742" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:25.026659 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:25.026772 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:25.027355 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8c39014a0ad4bf69adf32457e6906ca: Sending DeleteTablet for 3 replicas of tablet 3c855ea38a0e4642851b925d4479371f
I20260926 20:56:25.027658 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c855ea38a0e4642851b925d4479371f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43410
I20260926 20:56:25.027769 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c855ea38a0e4642851b925d4479371f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51548
I20260926 20:56:25.027861 8440 tablet_replica.cc:330] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0: stopping tablet replica
I20260926 20:56:25.027842 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 7978
I20260926 20:56:25.027916 8440 raft_consensus.cc:2229] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:25.027961 8440 raft_consensus.cc:2258] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:25.028023 8442 tablet_replica.cc:330] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742: stopping tablet replica
I20260926 20:56:25.028074 8442 raft_consensus.cc:2229] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:25.028106 8442 raft_consensus.cc:2258] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:25.029241 8442 ts_tablet_manager.cc:1905] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:25.029290 8440 ts_tablet_manager.cc:1905] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:25.030876 8440 ts_tablet_manager.cc:1918] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:56:25.030949 8440 log.cc:1196] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/3c855ea38a0e4642851b925d4479371f
I20260926 20:56:25.031152 8440 ts_tablet_manager.cc:1939] T 3c855ea38a0e4642851b925d4479371f P 78209d3846f8467bb3fd656dffeb72c0: Deleting consensus metadata
I20260926 20:56:25.031368 8442 ts_tablet_manager.cc:1918] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:56:25.031427 8442 log.cc:1196] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/3c855ea38a0e4642851b925d4479371f
I20260926 20:56:25.031606 8442 ts_tablet_manager.cc:1939] T 3c855ea38a0e4642851b925d4479371f P f79ca367d62b46bdb32256e3cfcf5742: Deleting consensus metadata
I20260926 20:56:25.031929 7893 catalog_manager.cc:4841] TS f79ca367d62b46bdb32256e3cfcf5742 (127.0.98.2:35471): tablet 3c855ea38a0e4642851b925d4479371f (table test-table [id=29c5f3d65e6746608f6059464d037fda]) successfully deleted
I20260926 20:56:25.032559 7892 catalog_manager.cc:4841] TS 78209d3846f8467bb3fd656dffeb72c0 (127.0.98.3:38091): tablet 3c855ea38a0e4642851b925d4479371f (table test-table [id=29c5f3d65e6746608f6059464d037fda]) successfully deleted
W20260926 20:56:25.034971 7893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39217 (error 108)
I20260926 20:56:25.035144 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 8118
I20260926 20:56:25.036120 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:25.035935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39217 (local address 127.0.0.1:35754)
0926 20:56:25.036042 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:25.036085 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":52,"threads_started":1}
W20260926 20:56:25.036281 7893 catalog_manager.cc:4584] TS 0552e62e93104b9488582a9b72bc0c35 (127.0.98.1:39217): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c855ea38a0e4642851b925d4479371f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39217: connect: Connection refused (error 111)
I20260926 20:56:25.042096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 8258
I20260926 20:56:25.047060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 7874
2026-09-26T20:56:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2604 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:56:25 dist-test-slave-fl7g krb5kdc[8503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:56:25 dist-test-slave-fl7g krb5kdc[8503](info): set up 2 sockets
Sep 26 20:56:25 dist-test-slave-fl7g krb5kdc[8503](info): commencing operation
krb5kdc: starting...
W20260926 20:56:26.109103 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:56:26 dist-test-slave-fl7g krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:56:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:56:26.235219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39157
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:26.318503 8516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:26.318773 8516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:26.318813 8516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:26.320052 8516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:26.320127 8516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:26.320149 8516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:26.320171 8516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:26.320199 8516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:26.321815 8516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:26.322105 8516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:26.322345 8516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:26.327808 8524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:26.327819 8521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:26.327898 8516 server_base.cc:1034] running on GCE node
W20260926 20:56:26.327823 8522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:26.328310 8516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:26.328699 8516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:26.329890 8516 hybrid_clock.cc:648] HybridClock initialized: now 1790456186329870 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T20:56:26Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1295 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:56:26 dist-test-slave-fl7g krb5kdc[8529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:56:26 dist-test-slave-fl7g krb5kdc[8529](info): set up 2 sockets
Sep 26 20:56:26 dist-test-slave-fl7g krb5kdc[8529](info): commencing operation
krb5kdc: starting...
W20260926 20:56:27.401118 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:56:27 dist-test-slave-fl7g krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:56:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:56:27.528211 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36089
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:27.612175 8542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:27.612402 8542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:27.612509 8542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:27.613695 8542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:27.613734 8542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:27.613749 8542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:27.613762 8542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:27.613775 8542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:27.615262 8542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:27.615468 8542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:27.615653 8542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:56:27.620640 8542 server_base.cc:1034] running on GCE node
W20260926 20:56:27.620527 8548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:27.620561 8550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:27.620537 8547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:27.621138 8542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:27.621454 8542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:27.622668 8542 hybrid_clock.cc:648] HybridClock initialized: now 1790456187622652 us; error 43 us; skew 500 ppm
Sep 26 20:56:27 dist-test-slave-fl7g krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:27.625255 8542 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:56:27.625711 8542 webserver.cc:460] Webserver started at http://127.0.98.62:45027/ using document root <none> and password file <none>
I20260926 20:56:27.625905 8542 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:27.625946 8542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:27.626046 8542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:27.627535 8542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "de2639a70d9f4077adf4a4912eb86318"
format_stamp: "Formatted at 2026-09-26 20:56:27 on dist-test-slave-fl7g"
server_key: "b5fd8c7cde56781443c87ae84cbc37bd"
server_key_iv: "6deca9172998e8c762ec781f44efeb1f"
server_key_version: "encryptionkey@0"
I20260926 20:56:27.627846 8542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "de2639a70d9f4077adf4a4912eb86318"
format_stamp: "Formatted at 2026-09-26 20:56:27 on dist-test-slave-fl7g"
server_key: "b5fd8c7cde56781443c87ae84cbc37bd"
server_key_iv: "6deca9172998e8c762ec781f44efeb1f"
server_key_version: "encryptionkey@0"
I20260926 20:56:27.629207 8542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:56:27.629920 8557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:27.630108 8542 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:27.630203 8542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "de2639a70d9f4077adf4a4912eb86318"
format_stamp: "Formatted at 2026-09-26 20:56:27 on dist-test-slave-fl7g"
server_key: "b5fd8c7cde56781443c87ae84cbc37bd"
server_key_iv: "6deca9172998e8c762ec781f44efeb1f"
server_key_version: "encryptionkey@0"
I20260926 20:56:27.630281 8542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:27.649794 8542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:27.651726 8542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:27.651876 8542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:27.656070 8542 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36089
I20260926 20:56:27.656111 8609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36089 every 8 connection(s)
I20260926 20:56:27.656689 8542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:27.657410 8610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:27.659818 8610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318: Bootstrap starting.
I20260926 20:56:27.660521 8610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:27.660807 8610 log.cc:824] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:27.661549 8610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318: No bootstrap required, opened a new log
I20260926 20:56:27.663156 8610 raft_consensus.cc:348] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2639a70d9f4077adf4a4912eb86318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36089 } }
I20260926 20:56:27.663337 8610 raft_consensus.cc:374] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:27.663404 8610 raft_consensus.cc:729] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2639a70d9f4077adf4a4912eb86318, State: Initialized, Role: FOLLOWER
I20260926 20:56:27.663607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 8542
I20260926 20:56:27.663712 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:56:27.663606 8610 consensus_queue.cc:260] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2639a70d9f4077adf4a4912eb86318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36089 } }
I20260926 20:56:27.663813 8610 raft_consensus.cc:388] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:27.663869 8610 raft_consensus.cc:482] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:27.663931 8610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:27.663936 392 external_mini_cluster.cc:1428] Setting key 9fd7a656f47c523e69e250c266961d97
I20260926 20:56:27.665138 8610 raft_consensus.cc:504] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2639a70d9f4077adf4a4912eb86318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36089 } }
I20260926 20:56:27.665266 8610 leader_election.cc:302] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de2639a70d9f4077adf4a4912eb86318; no voters:
I20260926 20:56:27.665555 8610 leader_election.cc:288] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:27.665490 8615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:27.665745 8615 raft_consensus.cc:686] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [term 1 LEADER]: Becoming Leader. State: Replica: de2639a70d9f4077adf4a4912eb86318, State: Running, Role: LEADER
I20260926 20:56:27.665889 8610 sys_catalog.cc:564] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:27.665911 8615 consensus_queue.cc:237] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2639a70d9f4077adf4a4912eb86318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36089 } }
I20260926 20:56:27.666450 8617 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de2639a70d9f4077adf4a4912eb86318. Latest consensus state: current_term: 1 leader_uuid: "de2639a70d9f4077adf4a4912eb86318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2639a70d9f4077adf4a4912eb86318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36089 } } }
I20260926 20:56:27.666549 8617 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:27.666777 8616 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de2639a70d9f4077adf4a4912eb86318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2639a70d9f4077adf4a4912eb86318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36089 } } }
I20260926 20:56:27.667093 8616 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:27.667325 8624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:27.668033 8624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:27.669620 8624 catalog_manager.cc:1347] Generated new cluster ID: 0e6c2cdcebc745f8aab799e0d7c530bb
I20260926 20:56:27.669673 8624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:56:27 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:27.676608 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:27.664515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.0.98.62:36089)
0926 20:56:27.664744 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:27.664755 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:27.664793 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:56:27.664909 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:27.664915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:27.664960 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:27.665122 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:27.665902 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:27.667286 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:27.668381 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:27.669065 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:27.669183 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:27.672117 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:27.672122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:27.672122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:27.672136 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:27.674794 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:27.675221 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:27.675225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:27.675225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:27.675360 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:27.675569 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:27.675570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:27.675570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:27.675920 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:27.676111 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:27.676198 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 20:56:27.676351 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":125,"threads_started":1}
I20260926 20:56:27.687858 8624 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:27.688504 8624 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:27.693830 8624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318: Generated new TSK 0
I20260926 20:56:27.694048 8624 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:56:27.724087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36089
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:27.808008 8636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:27.808223 8636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:27.808246 8636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:27.809527 8636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:27.809577 8636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:27.809605 8636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:56:27.811106 8636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:27.811355 8636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:27.811597 8636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:27.816505 8642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:27.816540 8644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:27.816558 8641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:27.816649 8636 server_base.cc:1034] running on GCE node
I20260926 20:56:27.816937 8636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:27.817246 8636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:27.818409 8636 hybrid_clock.cc:648] HybridClock initialized: now 1790456187818401 us; error 31 us; skew 500 ppm
Sep 26 20:56:27 dist-test-slave-fl7g krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:27.820871 8636 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:56:27.821333 8636 webserver.cc:460] Webserver started at http://127.0.98.1:33175/ using document root <none> and password file <none>
I20260926 20:56:27.821559 8636 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:27.821622 8636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:27.821772 8636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:27.822827 8636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "57b91ce226f04644a4b9b74b12ce50aa"
format_stamp: "Formatted at 2026-09-26 20:56:27 on dist-test-slave-fl7g"
server_key: "57d3d1567fd9d4acbc8b58d046007934"
server_key_iv: "3687a58f0c1b31fa41a0f6e945b1de79"
server_key_version: "encryptionkey@0"
I20260926 20:56:27.823199 8636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "57b91ce226f04644a4b9b74b12ce50aa"
format_stamp: "Formatted at 2026-09-26 20:56:27 on dist-test-slave-fl7g"
server_key: "57d3d1567fd9d4acbc8b58d046007934"
server_key_iv: "3687a58f0c1b31fa41a0f6e945b1de79"
server_key_version: "encryptionkey@0"
I20260926 20:56:27.824544 8636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:56:27.825304 8651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:27.825490 8636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:27.825582 8636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "57b91ce226f04644a4b9b74b12ce50aa"
format_stamp: "Formatted at 2026-09-26 20:56:27 on dist-test-slave-fl7g"
server_key: "57d3d1567fd9d4acbc8b58d046007934"
server_key_iv: "3687a58f0c1b31fa41a0f6e945b1de79"
server_key_version: "encryptionkey@0"
I20260926 20:56:27.825663 8636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:27.833725 8636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:27.835803 8636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:27.835969 8636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:27.836220 8636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:27.836578 8636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:27.836628 8636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:27.836673 8636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:27.836701 8636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:27.842943 8636 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39219
I20260926 20:56:27.842983 8764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39219 every 8 connection(s)
I20260926 20:56:27.843327 8636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:56:27.849301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 8636
I20260926 20:56:27.849436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:56:27.849666 392 external_mini_cluster.cc:1428] Setting key 7df9fb7c55f3fe8696a172fa6c2a531e
Sep 26 20:56:27 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:27.853807 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:27.844128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60073 (local address 127.0.98.62:36089)
0926 20:56:27.844188 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:27.844190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:27.844438 (+ 248us) server_negotiation.cc:407] Connection header received
0926 20:56:27.845423 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:27.845425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:27.845443 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:27.845494 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:27.846132 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:27.846639 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:27.847803 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:27.848017 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:27.848104 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:27.850661 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:27.850662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:27.850663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:27.850670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:27.852316 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:27.852802 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:27.852803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:27.852803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:27.852854 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:27.853178 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:27.853178 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:27.853179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:27.853281 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:27.853331 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:27.853675 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 20:56:27.853742 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 20:56:27.854113 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:27.844119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:60073)
0926 20:56:27.844285 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:27.844323 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:27.845303 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:27.845569 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:27.845573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:27.845840 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:27.846033 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:27.846037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:27.846746 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:27.846748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:27.847671 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:27.847675 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:27.848115 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:27.848116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:27.848244 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:27.848795 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:27.848804 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:27.850540 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:27.852458 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:27.852489 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:27.852491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:27.852724 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:27.852945 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:27.852946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:27.852947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:27.853108 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:27.853339 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:27.853344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:27.853524 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 20:56:27.853632 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 20:56:27.853814 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260926 20:56:27.854734 8765 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36089
I20260926 20:56:27.854844 8765 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:27.855054 8765 heartbeater.cc:511] Master 127.0.98.62:36089 requested a full tablet report, sending...
I20260926 20:56:27.855587 8574 ts_manager.cc:194] Registered new tserver with Master: 57b91ce226f04644a4b9b74b12ce50aa (127.0.98.1:39219)
I20260926 20:56:27.856266 8574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60073
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:56:27.878017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36089
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:27.959321 8770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:27.959592 8770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:27.959638 8770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:27.960865 8770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:27.960933 8770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:27.960978 8770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:56:27.962497 8770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:27.962777 8770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:27.963085 8770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:27.968760 8776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:27.968752 8775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:27.969108 8770 server_base.cc:1034] running on GCE node
W20260926 20:56:27.968752 8778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:27.970865 8770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:27.971176 8770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:27.972359 8770 hybrid_clock.cc:648] HybridClock initialized: now 1790456187972343 us; error 41 us; skew 500 ppm
Sep 26 20:56:27 dist-test-slave-fl7g krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:27.975005 8770 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:56:27.975471 8770 webserver.cc:460] Webserver started at http://127.0.98.2:35197/ using document root <none> and password file <none>
I20260926 20:56:27.975678 8770 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:27.975719 8770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:27.975811 8770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:27.976912 8770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "e7e7057bedc848af9116a800b38979ae"
format_stamp: "Formatted at 2026-09-26 20:56:27 on dist-test-slave-fl7g"
server_key: "1aa0ad68bc4b9f1b35b37de537e0f75c"
server_key_iv: "70a4cc010b17981712502d06e63ef21d"
server_key_version: "encryptionkey@0"
I20260926 20:56:27.977226 8770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e7e7057bedc848af9116a800b38979ae"
format_stamp: "Formatted at 2026-09-26 20:56:27 on dist-test-slave-fl7g"
server_key: "1aa0ad68bc4b9f1b35b37de537e0f75c"
server_key_iv: "70a4cc010b17981712502d06e63ef21d"
server_key_version: "encryptionkey@0"
I20260926 20:56:27.978468 8770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:56:27.979135 8785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:27.979287 8770 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:27.979368 8770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "e7e7057bedc848af9116a800b38979ae"
format_stamp: "Formatted at 2026-09-26 20:56:27 on dist-test-slave-fl7g"
server_key: "1aa0ad68bc4b9f1b35b37de537e0f75c"
server_key_iv: "70a4cc010b17981712502d06e63ef21d"
server_key_version: "encryptionkey@0"
I20260926 20:56:27.979423 8770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:27.988010 8770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:27.990248 8770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:27.990388 8770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:27.990572 8770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:27.990906 8770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:27.990944 8770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:27.990985 8770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:27.991011 8770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:27.996675 8770 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45345
I20260926 20:56:27.996744 8898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45345 every 8 connection(s)
I20260926 20:56:27.997064 8770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:56:28.003109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 8770
I20260926 20:56:28.003211 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
Sep 26 20:56:27 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:28.003427 392 external_mini_cluster.cc:1428] Setting key 308a87429661b5311f9957cf1dcadd76
I20260926 20:56:28.006852 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:27.997826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58019 (local address 127.0.98.62:36089)
0926 20:56:27.997884 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:27.997886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:27.998090 (+ 204us) server_negotiation.cc:407] Connection header received
0926 20:56:27.999012 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:27.999013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:27.999025 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:27.999084 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:27.999704 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.000246 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.001283 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.001471 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.001531 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.003745 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:28.003747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:28.003748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:28.003756 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:28.005402 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:28.005863 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:28.005865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:28.005865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:28.005929 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:28.006185 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:28.006186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:28.006187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:28.006308 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:28.006376 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:28.006661 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 20:56:28.006756 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 20:56:28.007062 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:27.997811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:58019)
0926 20:56:27.997969 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:27.998005 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:27.998912 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:27.999154 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:27.999157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:27.999410 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:27.999596 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:27.999600 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.000331 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.000332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.001209 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.001212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.001551 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.001552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.001664 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.002154 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:28.002162 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:28.003649 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:28.005508 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:28.005517 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:28.005519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:28.005770 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:28.005994 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:28.005995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:28.005995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:28.006117 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:28.006381 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:28.006387 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:28.006534 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 20:56:28.006649 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:56:28.006817 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260926 20:56:28.007579 8899 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36089
I20260926 20:56:28.007684 8899 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:28.007867 8899 heartbeater.cc:511] Master 127.0.98.62:36089 requested a full tablet report, sending...
I20260926 20:56:28.008266 8574 ts_manager.cc:194] Registered new tserver with Master: e7e7057bedc848af9116a800b38979ae (127.0.98.2:45345)
I20260926 20:56:28.008895 8574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58019
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:56:28.031596 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36089
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:28.117452 8904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:28.117672 8904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:28.117697 8904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:28.118882 8904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:28.118925 8904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:28.118952 8904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:56:28.120474 8904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:28.120725 8904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:28.120954 8904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:28.126052 8910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:28.126118 8904 server_base.cc:1034] running on GCE node
W20260926 20:56:28.126052 8909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:28.126048 8912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:28.126434 8904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:28.126693 8904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:28.127892 8904 hybrid_clock.cc:648] HybridClock initialized: now 1790456188127881 us; error 34 us; skew 500 ppm
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:28.130590 8904 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:56:28.131072 8904 webserver.cc:460] Webserver started at http://127.0.98.3:45147/ using document root <none> and password file <none>
I20260926 20:56:28.131270 8904 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:28.131311 8904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:28.131407 8904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:28.132503 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "506fd09c6b3a4b7b909a014ff45125be"
format_stamp: "Formatted at 2026-09-26 20:56:28 on dist-test-slave-fl7g"
server_key: "355ba7d77fecfe73ad77d083c283eb56"
server_key_iv: "55e1e97179943a5825c2f83df4215147"
server_key_version: "encryptionkey@0"
I20260926 20:56:28.132936 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "506fd09c6b3a4b7b909a014ff45125be"
format_stamp: "Formatted at 2026-09-26 20:56:28 on dist-test-slave-fl7g"
server_key: "355ba7d77fecfe73ad77d083c283eb56"
server_key_iv: "55e1e97179943a5825c2f83df4215147"
server_key_version: "encryptionkey@0"
I20260926 20:56:28.134335 8904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:56:28.135088 8919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:28.135306 8904 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:28.135413 8904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "506fd09c6b3a4b7b909a014ff45125be"
format_stamp: "Formatted at 2026-09-26 20:56:28 on dist-test-slave-fl7g"
server_key: "355ba7d77fecfe73ad77d083c283eb56"
server_key_iv: "55e1e97179943a5825c2f83df4215147"
server_key_version: "encryptionkey@0"
I20260926 20:56:28.135488 8904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:28.164204 8904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:28.166738 8904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:28.166859 8904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:28.167060 8904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:28.167384 8904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:28.167436 8904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:28.167524 8904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:28.167557 8904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:28.173385 8904 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39253
I20260926 20:56:28.173424 9032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39253 every 8 connection(s)
I20260926 20:56:28.173748 8904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:56:28.177346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 8904
I20260926 20:56:28.177469 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:56:28.177639 392 external_mini_cluster.cc:1428] Setting key 1f718dfd55c6d459875dfaa9e8a9c17c
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:28.183195 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:28.174419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57415 (local address 127.0.98.62:36089)
0926 20:56:28.174494 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:28.174497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:28.174685 (+ 188us) server_negotiation.cc:407] Connection header received
0926 20:56:28.175629 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:28.175630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:28.175640 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:28.175719 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:28.176347 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.176910 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.178109 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.178381 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.178446 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.180618 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:28.180619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:28.180619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:28.180625 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:28.181879 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:28.182306 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:28.182306 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:28.182307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:28.182358 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:28.182665 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:28.182666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:28.182666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:28.182764 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:28.182814 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:28.183062 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 20:56:28.183120 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:56:28.183415 9035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:28.174393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.3:57415)
0926 20:56:28.174567 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:28.174603 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:28.175529 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:28.175808 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:28.175811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:28.176070 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:28.176279 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.176283 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.176991 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.176992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.178023 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.178025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.178470 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.178472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.178607 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.179058 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:28.179068 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:28.180540 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:28.181966 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:28.181972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:28.181974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:28.182252 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:28.182435 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:28.182436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:28.182436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:28.182555 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:28.182817 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:28.182821 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:28.182940 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 20:56:28.183040 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 20:56:28.183162 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
I20260926 20:56:28.183883 9033 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36089
I20260926 20:56:28.183991 9033 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:28.184170 9033 heartbeater.cc:511] Master 127.0.98.62:36089 requested a full tablet report, sending...
I20260926 20:56:28.184585 8574 ts_manager.cc:194] Registered new tserver with Master: 506fd09c6b3a4b7b909a014ff45125be (127.0.98.3:39253)
I20260926 20:56:28.184911 8574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57415
I20260926 20:56:28.189350 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 20:56:28.263783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 8636
I20260926 20:56:28.268661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 8770
I20260926 20:56:28.273211 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 8904
I20260926 20:56:28.277578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 8542
I20260926 20:56:28.282855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36089
--webserver_interface=127.0.98.62
--webserver_port=45027
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:28.371899 9042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:28.372179 9042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:28.372221 9042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:28.373521 9042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:28.373593 9042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:28.373616 9042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:28.373646 9042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:28.373688 9042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:28.375286 9042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:28.375576 9042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:28.375816 9042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:28.380756 9048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:28.380793 9050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:28.380821 9042 server_base.cc:1034] running on GCE node
W20260926 20:56:28.380792 9047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:28.381222 9042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:28.381524 9042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:28.382680 9042 hybrid_clock.cc:648] HybridClock initialized: now 1790456188382663 us; error 48 us; skew 500 ppm
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:28.385308 9042 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:56:28.385813 9042 webserver.cc:460] Webserver started at http://127.0.98.62:45027/ using document root <none> and password file <none>
I20260926 20:56:28.386053 9042 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:28.386114 9042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:28.387530 9042 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:28.388322 9057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:28.388540 9042 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:28.388659 9042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "de2639a70d9f4077adf4a4912eb86318"
format_stamp: "Formatted at 2026-09-26 20:56:27 on dist-test-slave-fl7g"
server_key: "b5fd8c7cde56781443c87ae84cbc37bd"
server_key_iv: "6deca9172998e8c762ec781f44efeb1f"
server_key_version: "encryptionkey@0"
I20260926 20:56:28.389021 9042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:28.402457 9042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:28.404866 9042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:28.405018 9042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:28.409041 9042 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36089
I20260926 20:56:28.409076 9109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36089 every 8 connection(s)
I20260926 20:56:28.409608 9042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:28.410912 9110 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:56:28.411582 9110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318: Bootstrap starting.
I20260926 20:56:28.417930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 9042
I20260926 20:56:28.418258 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39219
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36089
--webserver_port=33175
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:56:28.419833 9110 log.cc:824] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:28.421918 9110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:56:28.422266 9110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318: Bootstrap complete.
I20260926 20:56:28.424733 9110 raft_consensus.cc:348] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2639a70d9f4077adf4a4912eb86318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36089 } }
I20260926 20:56:28.424980 9110 raft_consensus.cc:729] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2639a70d9f4077adf4a4912eb86318, State: Initialized, Role: FOLLOWER
I20260926 20:56:28.425163 9110 consensus_queue.cc:260] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2639a70d9f4077adf4a4912eb86318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36089 } }
I20260926 20:56:28.425240 9110 raft_consensus.cc:388] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:28.425292 9110 raft_consensus.cc:482] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:28.425343 9110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:56:28.426429 9110 raft_consensus.cc:504] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2639a70d9f4077adf4a4912eb86318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36089 } }
I20260926 20:56:28.426537 9110 leader_election.cc:302] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de2639a70d9f4077adf4a4912eb86318; no voters:
I20260926 20:56:28.426762 9110 leader_election.cc:288] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:56:28.426805 9114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:56:28.426920 9114 raft_consensus.cc:686] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [term 2 LEADER]: Becoming Leader. State: Replica: de2639a70d9f4077adf4a4912eb86318, State: Running, Role: LEADER
I20260926 20:56:28.427027 9114 consensus_queue.cc:237] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2639a70d9f4077adf4a4912eb86318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36089 } }
I20260926 20:56:28.427059 9110 sys_catalog.cc:564] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:28.427362 9115 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "de2639a70d9f4077adf4a4912eb86318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2639a70d9f4077adf4a4912eb86318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36089 } } }
I20260926 20:56:28.427534 9115 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:28.427836 9121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:28.427904 9116 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de2639a70d9f4077adf4a4912eb86318. Latest consensus state: current_term: 2 leader_uuid: "de2639a70d9f4077adf4a4912eb86318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2639a70d9f4077adf4a4912eb86318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36089 } } }
I20260926 20:56:28.428009 9116 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:28.428539 9121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:28.428864 9121 catalog_manager.cc:1259] Loaded cluster ID: 0e6c2cdcebc745f8aab799e0d7c530bb
I20260926 20:56:28.428905 9121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:56:28.430117 9121 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:28.430281 9121 catalog_manager.cc:5879] T 00000000000000000000000000000000 P de2639a70d9f4077adf4a4912eb86318: Loaded TSK: 0
I20260926 20:56:28.430487 9121 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:56:28.501808 9112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:28.502032 9112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:28.502058 9112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:28.503365 9112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:28.503414 9112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:28.503443 9112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:56:28.504966 9112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33175
--tserver_master_addrs=127.0.98.62:36089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:28.505230 9112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:28.505460 9112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:28.510372 9139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:28.510376 9137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:28.510753 9112 server_base.cc:1034] running on GCE node
W20260926 20:56:28.510491 9136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:28.511001 9112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:28.511241 9112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:28.512447 9112 hybrid_clock.cc:648] HybridClock initialized: now 1790456188512395 us; error 31 us; skew 500 ppm
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456188, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:28.514957 9112 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 20:56:28.515419 9112 webserver.cc:460] Webserver started at http://127.0.98.1:33175/ using document root <none> and password file <none>
I20260926 20:56:28.515651 9112 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:28.515705 9112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:28.517158 9112 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:28.517810 9146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:28.517998 9112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:28.518095 9112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "57b91ce226f04644a4b9b74b12ce50aa"
format_stamp: "Formatted at 2026-09-26 20:56:27 on dist-test-slave-fl7g"
server_key: "57d3d1567fd9d4acbc8b58d046007934"
server_key_iv: "3687a58f0c1b31fa41a0f6e945b1de79"
server_key_version: "encryptionkey@0"
I20260926 20:56:28.518456 9112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:28.538832 9112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:28.541162 9112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:28.541391 9112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:28.541644 9112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:28.542007 9112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:28.542064 9112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:28.542110 9112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:28.542130 9112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:28.549495 9112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39219
I20260926 20:56:28.549528 9259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39219 every 8 connection(s)
I20260926 20:56:28.549880 9112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:56:28.553813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 9112
I20260926 20:56:28.554144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45345
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36089
--webserver_port=35197
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:28.559630 9262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:28.550647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:49467)
0926 20:56:28.550851 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:28.550900 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:28.551490 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:28.552132 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:28.552136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:28.552763 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:28.552960 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.552965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.554646 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.554647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.555887 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.555890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.556613 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.556614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.556746 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.557302 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:28.557311 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:28.559462 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":88,"threads_started":1}
W20260926 20:56:28.559893 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:28.550681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49467 (local address 127.0.98.62:36089)
0926 20:56:28.551702 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:28.551711 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:28.551744 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:56:28.551865 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:28.551871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:28.551922 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:28.552057 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:28.553095 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.554542 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.556003 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.556497 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.556637 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.559670 (+ 3033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49467: BlockingRecv error: recv got EOF from 127.0.98.1:49467 (error 108)
Metrics: {"server-negotiator.queue_time_us":968,"thread_start_us":89,"threads_started":1}
W20260926 20:56:28.559921 9260 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36089 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36089: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:28.565267 9262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:28.560279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:53663)
0926 20:56:28.560315 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:28.560327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:28.560372 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:28.560567 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:28.560567 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:28.560682 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:28.560726 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.560728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.561748 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.561749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.562242 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.562243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.562748 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.562749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.562797 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.563255 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:28.563260 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:28.565216 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":17}
W20260926 20:56:28.565380 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:28.560342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53663 (local address 127.0.98.62:36089)
0926 20:56:28.560387 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:28.560389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:28.560399 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:56:28.560458 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:28.560459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:28.560473 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:28.560548 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:28.560804 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.561649 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.562323 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.562669 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.562752 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.565317 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53663: BlockingRecv error: recv got EOF from 127.0.98.1:53663 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:28.569700 9262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:28.565598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:39177)
0926 20:56:28.565648 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:28.565657 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:28.565712 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:28.565844 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:28.565845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:28.565960 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:28.565986 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.565987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.566589 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.566590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.567072 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.567074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.567525 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.567526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.567590 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.567968 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:28.567973 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:28.569650 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 20:56:28.569839 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:28.565648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39177 (local address 127.0.98.62:36089)
0926 20:56:28.565703 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:28.565705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:28.565714 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:56:28.565774 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:28.565775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:28.565786 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:28.565840 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:28.566062 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.566509 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.567163 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.567432 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.567526 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.569747 (+ 2221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39177: BlockingRecv error: recv got EOF from 127.0.98.1:39177 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260926 20:56:28.569839 9260 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:56:28.638315 9264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:28.638545 9264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:28.638572 9264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:28.639820 9264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:28.639868 9264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:28.639896 9264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:56:28.641415 9264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35197
--tserver_master_addrs=127.0.98.62:36089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:28.641732 9264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:28.641978 9264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:28.647543 9270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:28.647591 9264 server_base.cc:1034] running on GCE node
W20260926 20:56:28.647560 9269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:28.647585 9272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:28.648015 9264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:28.648262 9264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:28.649425 9264 hybrid_clock.cc:648] HybridClock initialized: now 1790456188649405 us; error 38 us; skew 500 ppm
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456188, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:28.652064 9264 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 20:56:28.652629 9264 webserver.cc:460] Webserver started at http://127.0.98.2:35197/ using document root <none> and password file <none>
I20260926 20:56:28.652873 9264 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:28.652946 9264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:28.654358 9264 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:28.655177 9279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:28.655392 9264 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:28.655467 9264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "e7e7057bedc848af9116a800b38979ae"
format_stamp: "Formatted at 2026-09-26 20:56:27 on dist-test-slave-fl7g"
server_key: "1aa0ad68bc4b9f1b35b37de537e0f75c"
server_key_iv: "70a4cc010b17981712502d06e63ef21d"
server_key_version: "encryptionkey@0"
I20260926 20:56:28.655808 9264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:28.674991 9264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:28.677346 9264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:28.677493 9264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:28.677717 9264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:28.678013 9264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:28.678054 9264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:28.678081 9264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:28.678095 9264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:28.684967 9264 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45345
I20260926 20:56:28.685019 9392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45345 every 8 connection(s)
I20260926 20:56:28.685370 9264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:56:28.689481 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 9264
I20260926 20:56:28.689792 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39253
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36089
--webserver_port=45147
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:56:28.693239 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:28.686104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48721 (local address 127.0.98.62:36089)
0926 20:56:28.686149 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:28.686151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:28.686413 (+ 262us) server_negotiation.cc:407] Connection header received
0926 20:56:28.686962 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:28.686963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:28.686979 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:28.687020 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:28.688002 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.688548 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.689607 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.689809 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.689896 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.693178 (+ 3282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48721: BlockingRecv error: recv got EOF from 127.0.98.2:48721 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:56:28.693298 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:28.686106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:48721)
0926 20:56:28.686285 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:28.686325 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:28.686846 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:28.687119 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:28.687123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:28.687705 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:28.687900 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.687905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.688644 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.688645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.689489 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.689492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.689909 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.689910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.690038 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.690592 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:28.690600 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:28.693084 (+ 2484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
W20260926 20:56:28.693536 9393 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36089 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36089: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:28.698761 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:28.693805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:46445)
0926 20:56:28.693854 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:28.693865 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:28.693940 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:28.694091 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:28.694092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:28.694248 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:28.694299 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.694301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.695109 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.695110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.695637 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.695639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.696007 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.696008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.696074 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.696596 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:28.696602 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:28.698671 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 20:56:28.698815 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:28.693861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46445 (local address 127.0.98.62:36089)
0926 20:56:28.693928 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:28.693930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:28.693940 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:56:28.694011 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:28.694012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:28.694023 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:28.694086 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:28.694494 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.695000 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.695737 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.695933 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.696010 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.698748 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46445: BlockingRecv error: recv got EOF from 127.0.98.2:46445 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:28.703492 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:28.699078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:54001)
0926 20:56:28.699118 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:28.699137 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:28.699171 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:28.699368 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:28.699368 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:28.699502 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:28.699533 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.699534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.700166 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.700167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.700725 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.700727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.701113 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.701114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.701177 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.701775 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:28.701781 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:28.703405 (+ 1624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 20:56:28.703542 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:28.699169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54001 (local address 127.0.98.62:36089)
0926 20:56:28.699231 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:28.699233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:28.699240 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:56:28.699261 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:28.699262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:28.699271 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:28.699361 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:28.699607 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.700078 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.700816 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.701022 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.701107 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.703476 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54001: BlockingRecv error: recv got EOF from 127.0.98.2:54001 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 20:56:28.703645 9393 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:56:28.779182 9396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:28.779385 9396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:28.779409 9396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:28.780704 9396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:28.780750 9396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:28.780778 9396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:56:28.782254 9396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45147
--tserver_master_addrs=127.0.98.62:36089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:28.782614 9396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:28.782892 9396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:28.788242 9402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:28.788375 9396 server_base.cc:1034] running on GCE node
W20260926 20:56:28.788225 9401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:28.788276 9404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:28.788667 9396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:28.788939 9396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:28.790108 9396 hybrid_clock.cc:648] HybridClock initialized: now 1790456188790098 us; error 38 us; skew 500 ppm
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456188, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:28.792680 9396 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 20:56:28.793206 9396 webserver.cc:460] Webserver started at http://127.0.98.3:45147/ using document root <none> and password file <none>
I20260926 20:56:28.793463 9396 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:28.793548 9396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:28.794942 9396 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:28.795576 9411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:28.795760 9396 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:28.795861 9396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "506fd09c6b3a4b7b909a014ff45125be"
format_stamp: "Formatted at 2026-09-26 20:56:28 on dist-test-slave-fl7g"
server_key: "355ba7d77fecfe73ad77d083c283eb56"
server_key_iv: "55e1e97179943a5825c2f83df4215147"
server_key_version: "encryptionkey@0"
I20260926 20:56:28.796247 9396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:28.808135 9396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:28.810171 9396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:28.810356 9396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:28.810617 9396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:28.810935 9396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:28.810992 9396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:28.811035 9396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:28.811062 9396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:28.817282 9396 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39253
I20260926 20:56:28.817322 9524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39253 every 8 connection(s)
I20260926 20:56:28.817684 9396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:28.825755 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 9396
W20260926 20:56:28.825930 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:28.818486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51177 (local address 127.0.98.62:36089)
0926 20:56:28.818554 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:28.818556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:28.818896 (+ 340us) server_negotiation.cc:407] Connection header received
0926 20:56:28.819490 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:28.819491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:28.819504 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:28.819571 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:28.820657 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.821149 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.822336 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.822532 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.822630 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.825870 (+ 3240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51177: BlockingRecv error: recv got EOF from 127.0.98.3:51177 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:56:28.826007 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:28.818472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.3:51177)
0926 20:56:28.818683 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:28.818718 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:28.819370 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:28.819648 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:28.819652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:28.820243 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:28.820541 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.820559 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.821248 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.821249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.822227 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.822230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.822653 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.822654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.822778 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.823352 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:28.823360 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:28.825793 (+ 2433us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":92,"threads_started":1}
W20260926 20:56:28.826273 9525 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36089 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36089: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:28.832775 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:28.826569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.3:41779)
0926 20:56:28.826606 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:28.826617 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:28.826661 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:28.826985 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:28.826987 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:28.827160 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:28.827229 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.827232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.828337 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.828338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.829226 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.829229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.829804 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.829804 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.829874 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.830419 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:28.830424 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:28.832719 (+ 2295us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":19}
W20260926 20:56:28.833026 9529 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:28.826664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41779 (local address 127.0.98.62:36089)
0926 20:56:28.826792 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:28.826793 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:28.826819 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:56:28.826836 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:28.826837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:28.826845 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:28.826901 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:28.827343 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.828208 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.829342 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.829720 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.829795 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.832946 (+ 3151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41779: BlockingRecv error: recv got EOF from 127.0.98.3:41779 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":103,"threads_started":1}
I20260926 20:56:28.835505 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:28.826394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43646 (local address 127.0.98.62:36089)
0926 20:56:28.826433 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:28.826435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:28.826561 (+ 126us) server_negotiation.cc:407] Connection header received
0926 20:56:28.826657 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:28.826657 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:28.826670 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:28.826729 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:28.827555 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.828273 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.829418 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.829710 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.829795 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.831409 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:28.831421 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:28.831422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:28.831439 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:28.833796 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:28.834199 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:28.834202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:28.834203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:28.834302 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:28.834534 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:28.834535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:28.834536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:28.834935 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:28.835088 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:28.835262 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 20:56:28.835380 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:28 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:28.838063 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:28.833319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.3:56355)
0926 20:56:28.833377 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:28.833389 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:28.833436 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:28.833567 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:28.833568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:28.833717 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:28.833761 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.833763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.834624 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.834626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.835407 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:28.835410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.835876 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.835877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:28.835933 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.836469 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:28.836474 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:28.838007 (+ 1533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 20:56:28.838167 9529 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:28.833375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56355 (local address 127.0.98.62:36089)
0926 20:56:28.833413 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:28.833415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:28.833443 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:56:28.833504 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:28.833504 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:28.833514 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:28.833550 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:28.833826 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.834538 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.835498 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:28.835805 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:28.835877 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:28.838092 (+ 2215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56355: BlockingRecv error: recv got EOF from 127.0.98.3:56355 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 20:56:28.838217 9525 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 20:56:29 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:29 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:29.575666 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:29.570280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:59171)
0926 20:56:29.570427 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:29.570443 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:29.570497 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:29.570656 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:29.570656 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:29.570819 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:29.570871 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:29.570873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:29.571570 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:29.571571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:29.572127 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:29.572129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:29.572650 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:29.572651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:29.572693 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:29.573208 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:29.573212 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:29.575540 (+ 2328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
W20260926 20:56:29.575721 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:29.570355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59171 (local address 127.0.98.62:36089)
0926 20:56:29.570499 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:29.570506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:29.570517 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:56:29.570543 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:29.570544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:29.570554 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:29.570627 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:29.570964 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:29.571483 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:29.572244 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:29.572567 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:29.572643 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:29.575639 (+ 2996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59171: BlockingRecv error: recv got EOF from 127.0.98.1:59171 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
Sep 26 20:56:29 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:29 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:29.709442 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:29.704182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:46241)
0926 20:56:29.704326 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:29.704364 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:29.704468 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:29.704667 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:29.704668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:29.704862 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:29.704915 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:29.704917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:29.705690 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:29.705691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:29.706258 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:29.706260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:29.706665 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:29.706665 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:29.706727 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:29.707264 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:29.707268 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:29.709330 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
W20260926 20:56:29.709543 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:29.704326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46241 (local address 127.0.98.62:36089)
0926 20:56:29.704443 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:29.704445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:29.704458 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:56:29.704570 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:29.704570 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:29.704591 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:29.704660 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:29.705037 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:29.705584 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:29.706356 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:29.706563 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:29.706655 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:29.709475 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46241: BlockingRecv error: recv got EOF from 127.0.98.2:46241 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 20:56:29 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:29 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:29.843601 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:29.838747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.3:38675)
0926 20:56:29.838958 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:29.838972 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:29.839092 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:29.839294 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:29.839295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:29.839469 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:29.839541 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:29.839543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:29.840296 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:29.840297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:29.840913 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:29.840915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:29.841242 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:29.841243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:29.841281 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:29.841767 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:29.841771 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:29.843533 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":112,"threads_started":1}
W20260926 20:56:29.843711 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:29.838863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38675 (local address 127.0.98.62:36089)
0926 20:56:29.838962 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:29.838964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:29.839074 (+ 110us) server_negotiation.cc:407] Connection header received
0926 20:56:29.839202 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:29.839203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:29.839217 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:29.839277 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:29.839643 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:29.840225 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:29.841005 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:29.841180 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:29.841232 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:29.843642 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38675: BlockingRecv error: recv got EOF from 127.0.98.3:38675 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 20:56:30 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:30 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:30.581183 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:30.576367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:38219)
0926 20:56:30.576544 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:30.576556 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:30.576621 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:30.576785 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:30.576786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:30.576945 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:30.576994 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:30.576995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:30.577679 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:30.577681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:30.578208 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:30.578209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:30.578578 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:30.578579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:30.578617 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:30.579122 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:30.579127 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:30.581116 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":120,"threads_started":1}
W20260926 20:56:30.581382 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:30.576492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38219 (local address 127.0.98.62:36089)
0926 20:56:30.576645 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:30.576648 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:30.576658 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:56:30.576683 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:30.576684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:30.576694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:30.576776 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:30.577069 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:30.577599 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:30.578289 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:30.578479 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:30.578564 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:30.581283 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38219: BlockingRecv error: recv got EOF from 127.0.98.1:38219 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
Sep 26 20:56:30 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:30 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:30.714632 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:30.710003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:36283)
0926 20:56:30.710170 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:30.710182 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:30.710260 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:30.710399 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:30.710400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:30.710562 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:30.710611 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:30.710613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:30.711251 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:30.711252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:30.711771 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:30.711772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:30.712095 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:30.712096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:30.712135 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:30.712747 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:30.712751 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:30.714565 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":105,"threads_started":1}
W20260926 20:56:30.714738 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:30.710131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36283 (local address 127.0.98.62:36089)
0926 20:56:30.710194 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:30.710197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:30.710248 (+ 51us) server_negotiation.cc:407] Connection header received
0926 20:56:30.710308 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:30.710309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:30.710319 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:30.710376 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:30.710683 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:30.711180 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:30.711846 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:30.712027 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:30.712081 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:30.714670 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36283: BlockingRecv error: recv got EOF from 127.0.98.2:36283 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 20:56:30 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:30 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:30.849596 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:30.844360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.3:43065)
0926 20:56:30.844607 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:30.844621 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:30.844729 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:30.844907 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:30.844908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:30.845097 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:30.845167 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:30.845169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:30.845897 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:30.845899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:30.846441 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:30.846443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:30.846834 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:30.846835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:30.846904 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:30.847401 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:30.847406 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:30.849497 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":147,"threads_started":1}
W20260926 20:56:30.849864 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:30.844544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43065 (local address 127.0.98.62:36089)
0926 20:56:30.844627 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:30.844629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:30.844709 (+ 80us) server_negotiation.cc:407] Connection header received
0926 20:56:30.844802 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:30.844804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:30.844821 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:30.844899 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:30.845259 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:30.845783 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:30.846546 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:30.846746 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:30.846821 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:30.849641 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43065: BlockingRecv error: recv got EOF from 127.0.98.3:43065 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 20:56:31 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:31 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:31.590728 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:31.581722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:60123)
0926 20:56:31.581862 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:31.581881 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:31.581935 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:31.582147 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:31.582148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:31.582304 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:31.582350 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:31.582353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:31.583016 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:31.583017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:31.583530 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:31.583532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:31.583899 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:31.583900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:31.583939 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:31.588789 (+ 4850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:31.588796 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:31.590666 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
W20260926 20:56:31.590911 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:31.581818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60123 (local address 127.0.98.62:36089)
0926 20:56:31.581991 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:31.581993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:31.582011 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:56:31.582038 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:31.582038 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:31.582049 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:31.582126 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:31.582430 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:31.582931 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:31.583627 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:31.583829 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:31.583886 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:31.590828 (+ 6942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60123: BlockingRecv error: recv got EOF from 127.0.98.1:60123 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":95,"threads_started":1}
Sep 26 20:56:31 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:31 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:31.720132 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:31.715232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:56001)
0926 20:56:31.715368 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:31.715386 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:31.715472 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:31.715690 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:31.715691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:31.715866 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:31.715913 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:31.715915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:31.716668 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:31.716670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:31.717218 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:31.717219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:31.717596 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:31.717597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:31.717664 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:31.718129 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:31.718133 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:31.720040 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
W20260926 20:56:31.720211 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:31.715368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56001 (local address 127.0.98.62:36089)
0926 20:56:31.715443 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:31.715445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:31.715457 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:56:31.715569 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:31.715570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:31.715590 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:31.715658 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:31.715994 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:31.716582 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:31.717306 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:31.717506 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:31.717595 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:31.720155 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56001: BlockingRecv error: recv got EOF from 127.0.98.2:56001 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 20:56:31 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:31 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:31.855075 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:31.850152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.3:45745)
0926 20:56:31.850297 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:31.850310 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:31.850377 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:31.850557 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:31.850558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:31.850741 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:31.850790 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:31.850792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:31.851461 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:31.851462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:31.852137 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:31.852139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:31.852547 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:31.852548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:31.852608 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:31.853131 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:31.853136 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:31.854980 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
W20260926 20:56:31.855136 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:31.850280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45745 (local address 127.0.98.62:36089)
0926 20:56:31.850349 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:31.850352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:31.850363 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:56:31.850453 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:31.850454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:31.850475 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:31.850548 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:31.850874 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:31.851373 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:31.852222 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:31.852451 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:31.852545 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:31.855073 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45745: BlockingRecv error: recv got EOF from 127.0.98.3:45745 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 20:56:32 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:32 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:32.596671 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:32.591439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:46117)
0926 20:56:32.591638 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:32.591658 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:32.591729 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:32.591914 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:32.591915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:32.592084 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:32.592134 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:32.592136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:32.592990 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:32.592991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:32.593545 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:32.593547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:32.593927 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:32.593928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:32.593984 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:32.594548 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:32.594555 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:32.596609 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":95,"threads_started":1}
W20260926 20:56:32.596796 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:32.591542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46117 (local address 127.0.98.62:36089)
0926 20:56:32.591677 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:32.591679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:32.591712 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:56:32.591779 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:32.591780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:32.591792 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:32.591867 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:32.592234 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:32.592884 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:32.593619 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:32.593844 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:32.593897 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:32.596710 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46117: BlockingRecv error: recv got EOF from 127.0.98.1:46117 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
Sep 26 20:56:32 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:32 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:32.725502 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:32.720773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:58607)
0926 20:56:32.720928 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:32.720939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:32.721029 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:32.721202 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:32.721203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:32.721364 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:32.721412 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:32.721414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:32.722095 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:32.722096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:32.722649 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:32.722651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:32.723001 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:32.723002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:32.723044 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:32.723605 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:32.723610 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:32.725438 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
W20260926 20:56:32.725575 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:32.720840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58607 (local address 127.0.98.62:36089)
0926 20:56:32.720928 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:32.720931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:32.721017 (+ 86us) server_negotiation.cc:407] Connection header received
0926 20:56:32.721097 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:32.721097 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:32.721110 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:32.721192 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:32.721479 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:32.722005 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:32.722746 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:32.722932 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:32.722986 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:32.725528 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58607: BlockingRecv error: recv got EOF from 127.0.98.2:58607 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 20:56:32 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:32 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:32.860718 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:32.855712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.3:50105)
0926 20:56:32.855966 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:32.855980 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:32.856083 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:32.856271 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:32.856272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:32.856483 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:32.856532 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:32.856533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:32.857194 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:32.857195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:32.857748 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:32.857749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:32.858154 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:32.858154 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:32.858225 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:32.858715 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:32.858720 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:32.860648 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":95,"threads_started":1}
W20260926 20:56:32.860850 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:32.855848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50105 (local address 127.0.98.62:36089)
0926 20:56:32.855950 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:32.855952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:32.856063 (+ 111us) server_negotiation.cc:407] Connection header received
0926 20:56:32.856164 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:32.856165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:32.856196 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:32.856266 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:32.856612 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:32.857107 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:32.857843 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:32.858081 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:32.858155 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:32.860768 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50105: BlockingRecv error: recv got EOF from 127.0.98.3:50105 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 20:56:33 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:33 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:33.602172 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:33.597198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:58279)
0926 20:56:33.597359 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:33.597369 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:33.597434 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:33.597588 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:33.597589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:33.597754 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:33.597804 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:33.597805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:33.598534 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:33.598535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:33.599125 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:33.599127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:33.599500 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:33.599501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:33.599560 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:33.600207 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:33.600212 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:33.602108 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
W20260926 20:56:33.602361 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:33.597291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58279 (local address 127.0.98.62:36089)
0926 20:56:33.597392 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:33.597394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:33.597417 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:56:33.597494 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:33.597494 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:33.597506 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:33.597573 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:33.597909 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:33.598426 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:33.599244 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:33.599438 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:33.599502 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:33.602268 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58279: BlockingRecv error: recv got EOF from 127.0.98.1:58279 (error 108)
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":51,"threads_started":1}
Sep 26 20:56:33 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:33 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:33.730916 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:33.726120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:35453)
0926 20:56:33.726417 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:33.726429 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:33.726515 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:33.726693 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:33.726694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:33.726931 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:33.726977 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:33.726979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:33.727703 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:33.727703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:33.728303 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:33.728305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:33.728632 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:33.728633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:33.728697 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:33.729136 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:33.729140 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:33.730858 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":100,"threads_started":1}
W20260926 20:56:33.731027 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:33.726275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35453 (local address 127.0.98.62:36089)
0926 20:56:33.726355 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:33.726357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:33.726571 (+ 214us) server_negotiation.cc:407] Connection header received
0926 20:56:33.726598 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:33.726598 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:33.726613 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:33.726672 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:33.727043 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:33.727594 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:33.728365 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:33.728568 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:33.728622 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:33.730956 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35453: BlockingRecv error: recv got EOF from 127.0.98.2:35453 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 20:56:33 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:33 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:33.866160 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:33.861209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.3:51215)
0926 20:56:33.861368 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:33.861384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:33.861443 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:33.861613 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:33.861614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:33.861789 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:33.861840 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:33.861842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:33.862548 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:33.862549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:33.863113 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:33.863116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:33.863519 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:33.863521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:33.863579 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:33.864071 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:33.864077 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:33.866100 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
W20260926 20:56:33.866348 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:33.861274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51215 (local address 127.0.98.62:36089)
0926 20:56:33.861338 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:33.861340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:33.861442 (+ 102us) server_negotiation.cc:407] Connection header received
0926 20:56:33.861486 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:33.861487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:33.861498 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:33.861553 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:33.861949 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:33.862452 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:33.863207 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:33.863402 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:33.863462 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:33.866274 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51215: BlockingRecv error: recv got EOF from 127.0.98.3:51215 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 20:56:34 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:34 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:34.607642 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:34.602690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:47103)
0926 20:56:34.602834 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:34.602847 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:34.602906 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:34.603138 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:34.603140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:34.603303 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:34.603352 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:34.603353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:34.604096 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:34.604097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:34.604671 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:34.604672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:34.605028 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:34.605029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:34.605072 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:34.605673 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:34.605678 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:34.607569 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
W20260926 20:56:34.607826 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:34.602797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47103 (local address 127.0.98.62:36089)
0926 20:56:34.602922 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:34.602930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:34.602940 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:56:34.602968 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:34.602968 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:34.602979 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:34.603075 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:34.603464 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:34.603993 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:34.604744 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:34.604947 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:34.605013 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:34.607729 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47103: BlockingRecv error: recv got EOF from 127.0.98.1:47103 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
Sep 26 20:56:34 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:34 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:34.736747 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:34.731529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:50491)
0926 20:56:34.731673 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:34.731685 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:34.731759 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:34.731903 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:34.731904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:34.732071 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:34.732121 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:34.732123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:34.732937 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:34.732939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:34.733480 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:34.733482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:34.733863 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:34.733864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:34.733919 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:34.734590 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:34.734600 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:34.736672 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
W20260926 20:56:34.736846 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:34.731603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50491 (local address 127.0.98.62:36089)
0926 20:56:34.731678 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:34.731680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:34.731767 (+ 87us) server_negotiation.cc:407] Connection header received
0926 20:56:34.731805 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:34.731806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:34.731818 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:34.731889 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:34.732194 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:34.732831 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:34.733565 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:34.733775 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:34.733832 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:34.736772 (+ 2940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50491: BlockingRecv error: recv got EOF from 127.0.98.2:50491 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 20:56:34 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:34 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:34.871935 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:34.866902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.3:38113)
0926 20:56:34.867071 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:34.867086 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:34.867184 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:34.867351 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:34.867352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:34.867572 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:34.867677 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:34.867680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:34.868360 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:34.868361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:34.868918 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:34.868920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:34.869278 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:34.869278 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:34.869317 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:34.869795 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:34.869800 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:34.871875 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
W20260926 20:56:34.872108 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:34.866986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38113 (local address 127.0.98.62:36089)
0926 20:56:34.867105 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:34.867107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:34.867152 (+ 45us) server_negotiation.cc:407] Connection header received
0926 20:56:34.867258 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:34.867258 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:34.867275 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:34.867342 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:34.867768 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:34.868284 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:34.869014 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:34.869206 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:34.869263 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:34.872003 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38113: BlockingRecv error: recv got EOF from 127.0.98.3:38113 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 20:56:35 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:35 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:35.613059 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:35.608135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:43759)
0926 20:56:35.608300 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:35.608313 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:35.608392 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:35.608603 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:35.608604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:35.608772 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:35.608820 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:35.608822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:35.609510 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:35.609511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:35.610082 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:35.610084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:35.610467 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:35.610468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:35.610515 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:35.611040 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:35.611045 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:35.612999 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":70,"threads_started":1}
W20260926 20:56:35.613204 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:35.608246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43759 (local address 127.0.98.62:36089)
0926 20:56:35.608467 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:35.608474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:35.608490 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:56:35.608511 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:35.608511 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:35.608521 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:35.608586 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:35.608917 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:35.609435 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:35.610183 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:35.610387 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:35.610451 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:35.613105 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43759: BlockingRecv error: recv got EOF from 127.0.98.1:43759 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":73,"threads_started":1}
Sep 26 20:56:35 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:35 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:35.742717 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:35.737402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:57153)
0926 20:56:35.737557 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:35.737569 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:35.737630 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:35.737774 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:35.737775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:35.737938 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:35.737987 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:35.737989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:35.739024 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:35.739025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:35.739666 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:35.739668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:35.740065 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:35.740065 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:35.740108 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:35.740667 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:35.740673 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:35.742654 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
W20260926 20:56:35.742923 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:35.737470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57153 (local address 127.0.98.62:36089)
0926 20:56:35.737577 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:35.737579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:35.737616 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:56:35.737680 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:35.737681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:35.737693 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:35.737758 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:35.738092 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:35.738909 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:35.739777 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:35.739993 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:35.740060 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:35.742852 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57153: BlockingRecv error: recv got EOF from 127.0.98.2:57153 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 20:56:35 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:35 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:35.877710 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:35.872603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.3:47301)
0926 20:56:35.872762 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:35.872777 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:35.872854 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:35.873009 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:35.873010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:35.873192 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:35.873241 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:35.873243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:35.873926 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:35.873927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:35.874471 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:35.874473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:35.874841 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:35.874842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:35.874908 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:35.875443 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:35.875448 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:35.877644 (+ 2196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
W20260926 20:56:35.877943 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:35.872738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47301 (local address 127.0.98.62:36089)
0926 20:56:35.872829 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:35.872831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:35.872841 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:56:35.872932 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:35.872933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:35.872949 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:35.873003 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:35.873324 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:35.873834 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:35.874561 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:35.874754 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:35.874840 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:35.877866 (+ 3026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47301: BlockingRecv error: recv got EOF from 127.0.98.3:47301 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 20:56:36 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:36 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:36.625031 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:36.613754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:49809)
0926 20:56:36.613890 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:36.613902 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:36.613962 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:36.614239 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:36.614239 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:36.614412 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:36.614461 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:36.614463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:36.615188 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:36.615190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:36.615721 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:36.615722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:36.616100 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:36.616101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:36.616142 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:36.622942 (+ 6800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:36.622948 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:36.624968 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
W20260926 20:56:36.625213 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:36.613920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49809 (local address 127.0.98.62:36089)
0926 20:56:36.614078 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:36.614080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:36.614092 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:56:36.614128 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:36.614129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:36.614139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:36.614230 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:36.614554 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:36.615073 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:36.615807 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:36.616028 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:36.616086 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:36.625105 (+ 9019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49809: BlockingRecv error: recv got EOF from 127.0.98.1:49809 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
Sep 26 20:56:36 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:36 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:36.748260 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:36.743323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:36121)
0926 20:56:36.743499 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:36.743510 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:36.743574 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:36.743718 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:36.743718 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:36.743885 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:36.743935 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:36.743937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:36.744832 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:36.744833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:36.745365 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:36.745366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:36.745765 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:36.745766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:36.745802 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:36.746297 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:36.746301 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:36.748194 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
W20260926 20:56:36.748391 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:36.743385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36121 (local address 127.0.98.62:36089)
0926 20:56:36.743517 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:36.743519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:36.743560 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:56:36.743618 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:36.743619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:36.743630 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:36.743686 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:36.744014 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:36.744754 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:36.745442 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:36.745697 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:36.745758 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:36.748317 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36121: BlockingRecv error: recv got EOF from 127.0.98.2:36121 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 20:56:36 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:36 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:36.883088 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:36.878388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.3:54899)
0926 20:56:36.878510 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:36.878530 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:36.878583 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:36.878746 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:36.878747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:36.878929 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:36.878978 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:36.878980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:36.879645 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:36.879646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:36.880172 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:36.880173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:36.880515 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:36.880516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:36.880557 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:36.881108 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:36.881113 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:36.882967 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
W20260926 20:56:36.883168 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:36.878477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54899 (local address 127.0.98.62:36089)
0926 20:56:36.878571 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:36.878573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:36.878587 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:56:36.878680 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:36.878681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:36.878694 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:36.878740 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:36.879066 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:36.879549 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:36.880254 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:36.880462 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:36.880516 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:36.883092 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54899: BlockingRecv error: recv got EOF from 127.0.98.3:54899 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 20:56:37 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:37 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:37.630862 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:37.625567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:60109)
0926 20:56:37.625733 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:37.625744 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:37.625803 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:37.625957 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:37.625957 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:37.626127 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:37.626180 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:37.626181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:37.626833 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:37.626834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:37.627455 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:37.627458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:37.627995 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:37.627996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:37.628051 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:37.628665 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:37.628671 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:37.630762 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":121,"threads_started":1}
W20260926 20:56:37.630923 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:37.625675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60109 (local address 127.0.98.62:36089)
0926 20:56:37.625777 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:37.625779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:37.625791 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:56:37.625868 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:37.625869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:37.625880 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:37.625940 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:37.626241 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:37.626744 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:37.627642 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:37.627902 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:37.627973 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:37.630849 (+ 2876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60109: BlockingRecv error: recv got EOF from 127.0.98.1:60109 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":52,"threads_started":1}
Sep 26 20:56:37 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:37 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:37.753750 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:37.748846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:48591)
0926 20:56:37.749023 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:37.749042 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:37.749098 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:37.749254 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:37.749255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:37.749417 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:37.749464 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:37.749466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:37.750345 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:37.750347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:37.750953 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:37.750956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:37.751297 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:37.751298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:37.751335 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:37.751811 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:37.751815 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:37.753690 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
W20260926 20:56:37.753863 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:37.748975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48591 (local address 127.0.98.62:36089)
0926 20:56:37.749055 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:37.749057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:37.749083 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:56:37.749155 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:37.749162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:37.749175 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:37.749235 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:37.749533 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:37.750273 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:37.751038 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:37.751229 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:37.751285 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:37.753789 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48591: BlockingRecv error: recv got EOF from 127.0.98.2:48591 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 20:56:37 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:37 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:37.888507 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:37.883680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.3:58443)
0926 20:56:37.883888 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:37.883901 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:37.883997 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:37.884159 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:37.884161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:37.884347 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:37.884396 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:37.884397 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:37.885099 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:37.885100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:37.885631 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:37.885633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:37.886007 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:37.886008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:37.886065 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:37.886526 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:37.886530 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:37.888386 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":134,"threads_started":1}
W20260926 20:56:37.888562 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:37.883903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58443 (local address 127.0.98.62:36089)
0926 20:56:37.883989 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:37.883990 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:37.884000 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:56:37.884058 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:37.884058 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:37.884074 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:37.884151 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:37.884501 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:37.885005 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:37.885738 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:37.885925 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:37.885997 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:37.888496 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58443: BlockingRecv error: recv got EOF from 127.0.98.3:58443 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 20:56:38 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:38 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:38.636338 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:38.631408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:33769)
0926 20:56:38.631529 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:38.631540 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:38.631604 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:38.631817 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:38.631818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:38.632008 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:38.632054 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:38.632056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:38.632819 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:38.632820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:38.633354 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:38.633356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:38.633733 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:38.633734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:38.633790 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:38.634334 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:38.634339 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:38.636236 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
W20260926 20:56:38.636395 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:38.631466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33769 (local address 127.0.98.62:36089)
0926 20:56:38.631586 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:38.631588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:38.631599 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:56:38.631661 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:38.631662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:38.631677 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:38.631781 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:38.632146 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:38.632734 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:38.633448 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:38.633658 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:38.633735 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:38.636322 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33769: BlockingRecv error: recv got EOF from 127.0.98.1:33769 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
Sep 26 20:56:38 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:38 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:38.759297 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:38.754320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:52241)
0926 20:56:38.754537 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:38.754549 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:38.754648 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:38.754819 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:38.754820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:38.755014 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:38.755062 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:38.755064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:38.755768 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:38.755769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:38.756316 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:38.756318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:38.756740 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:38.756741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:38.756844 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:38.757357 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:38.757362 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:38.759239 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
W20260926 20:56:38.759421 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:38.754429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52241 (local address 127.0.98.62:36089)
0926 20:56:38.754533 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:38.754535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:38.754627 (+ 92us) server_negotiation.cc:407] Connection header received
0926 20:56:38.754716 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:38.754717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:38.754737 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:38.754800 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:38.755152 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:38.755672 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:38.756440 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:38.756652 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:38.756739 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:38.759339 (+ 2600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52241: BlockingRecv error: recv got EOF from 127.0.98.2:52241 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 20:56:38 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:38 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:38.894069 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:38.889113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.3:55259)
0926 20:56:38.889278 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:38.889299 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:38.889355 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:38.889535 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:38.889537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:38.889713 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:38.889762 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:38.889764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:38.890441 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:38.890443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:38.891037 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:38.891039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:38.891508 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:38.891509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:38.891554 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:38.892180 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:38.892185 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:38.894009 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
W20260926 20:56:38.894243 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:38.889262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55259 (local address 127.0.98.62:36089)
0926 20:56:38.889331 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:38.889332 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:38.889341 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:56:38.889418 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:38.889419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:38.889431 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:38.889492 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:38.889839 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:38.890356 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:38.891195 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:38.891441 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:38.891501 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:38.894140 (+ 2639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55259: BlockingRecv error: recv got EOF from 127.0.98.3:55259 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 20:56:39 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:39 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:39.642232 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:39.637017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:55077)
0926 20:56:39.637164 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:39.637185 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:39.637241 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:39.637416 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:39.637417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:39.637590 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:39.637638 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:39.637640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:39.638455 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:39.638456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:39.639238 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:39.639241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:39.639770 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:39.639771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:39.639823 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:39.640374 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:39.640379 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:39.642171 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
W20260926 20:56:39.642359 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:39.637125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55077 (local address 127.0.98.62:36089)
0926 20:56:39.637263 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:39.637265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:39.637280 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:56:39.637307 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:39.637307 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:39.637318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:39.637393 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:39.637736 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:39.638367 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:39.639329 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:39.639679 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:39.639767 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:39.642273 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55077: BlockingRecv error: recv got EOF from 127.0.98.1:55077 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
Sep 26 20:56:39 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:39 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:39.764757 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:39.759830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:52549)
0926 20:56:39.759947 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:39.759958 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:39.760026 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:39.760159 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:39.760160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:39.760326 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:39.760375 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:39.760377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:39.761134 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:39.761135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:39.761723 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:39.761725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:39.762076 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:39.762077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:39.762115 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:39.762623 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:39.762629 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:39.764698 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
W20260926 20:56:39.764912 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:39.759897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52549 (local address 127.0.98.62:36089)
0926 20:56:39.759968 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:39.759970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:39.760034 (+ 64us) server_negotiation.cc:407] Connection header received
0926 20:56:39.760093 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:39.760093 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:39.760109 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:39.760155 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:39.760547 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:39.761037 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:39.761815 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:39.762004 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:39.762073 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:39.764832 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52549: BlockingRecv error: recv got EOF from 127.0.98.2:52549 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 20:56:39 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:39 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:39.900090 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:39.894579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.3:50073)
0926 20:56:39.894883 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:39.894896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:39.895001 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:39.895164 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:39.895165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:39.895349 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:39.895396 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:39.895398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:39.896159 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:39.896160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:39.896786 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:39.896788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:39.897179 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:39.897180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:39.897240 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:39.897745 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:39.897750 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:39.899950 (+ 2200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":192,"threads_started":1}
W20260926 20:56:39.900113 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:39.894766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50073 (local address 127.0.98.62:36089)
0926 20:56:39.894859 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:39.894862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:39.894979 (+ 117us) server_negotiation.cc:407] Connection header received
0926 20:56:39.895074 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:39.895075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:39.895090 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:39.895148 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:39.895551 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:39.896071 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:39.896883 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:39.897086 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:39.897177 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:39.900064 (+ 2887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50073: BlockingRecv error: recv got EOF from 127.0.98.3:50073 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 20:56:40 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:40 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:40.647687 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:40.642909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:52377)
0926 20:56:40.643060 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:40.643078 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:40.643131 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:40.643291 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:40.643292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:40.643454 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:40.643501 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:40.643503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:40.644168 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:40.644169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:40.644839 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:40.644841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:40.645229 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:40.645230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:40.645270 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:40.645765 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:40.645770 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:40.647612 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
W20260926 20:56:40.647796 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:40.642995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52377 (local address 127.0.98.62:36089)
0926 20:56:40.643148 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:40.643150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:40.643162 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:56:40.643194 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:40.643194 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:40.643205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:40.643271 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:40.643570 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:40.644078 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:40.644938 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:40.645155 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:40.645221 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:40.647722 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52377: BlockingRecv error: recv got EOF from 127.0.98.1:52377 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
Sep 26 20:56:40 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:40 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:40.770117 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:40.765278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:53539)
0926 20:56:40.765443 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:40.765454 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:40.765513 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:40.765657 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:40.765658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:40.765824 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:40.765872 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:40.765873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:40.766516 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:40.766517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:40.767071 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:40.767073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:40.767502 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:40.767502 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:40.767545 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:40.768091 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:40.768101 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:40.770057 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
W20260926 20:56:40.770262 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:40.765392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53539 (local address 127.0.98.62:36089)
0926 20:56:40.765462 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:40.765464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:40.765499 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:56:40.765579 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:40.765580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:40.765591 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:40.765637 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:40.765938 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:40.766428 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:40.767227 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:40.767426 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:40.767491 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:40.770176 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53539: BlockingRecv error: recv got EOF from 127.0.98.2:53539 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 20:56:40 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:40 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:40.905820 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:40.900707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.3:46547)
0926 20:56:40.900846 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:40.900865 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:40.900929 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:40.901155 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:40.901165 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:40.901341 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:40.901415 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:40.901417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:40.902176 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:40.902177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:40.902712 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:40.902714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:40.903180 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:40.903181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:40.903285 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:40.903841 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:40.903845 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:40.905762 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
W20260926 20:56:40.905949 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:40.900791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46547 (local address 127.0.98.62:36089)
0926 20:56:40.900879 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:40.900881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:40.900938 (+ 57us) server_negotiation.cc:407] Connection header received
0926 20:56:40.901022 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:40.901023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:40.901036 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:40.901103 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:40.901528 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:40.902062 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:40.902825 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:40.903087 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:40.903185 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:40.905866 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46547: BlockingRecv error: recv got EOF from 127.0.98.3:46547 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 20:56:41 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:41 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:41.657124 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:41.648312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:47407)
0926 20:56:41.648499 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:41.648512 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:41.648601 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:41.648750 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:41.648751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:41.648918 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:41.648968 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:41.648970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:41.649630 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:41.649630 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:41.650145 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:41.650147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:41.650471 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:41.650471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:41.650517 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:41.655142 (+ 4625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:41.655148 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:41.657063 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":141,"threads_started":1}
W20260926 20:56:41.657300 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:41.648466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47407 (local address 127.0.98.62:36089)
0926 20:56:41.648566 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:41.648568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:41.648583 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:56:41.648655 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:41.648655 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:41.648667 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:41.648730 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:41.649041 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:41.649558 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:41.650220 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:41.650403 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:41.650452 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:41.657198 (+ 6746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47407: BlockingRecv error: recv got EOF from 127.0.98.1:47407 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":48,"threads_started":1}
Sep 26 20:56:41 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:41 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:41.775862 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:41.770853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:39943)
0926 20:56:41.771048 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:41.771062 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:41.771138 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:41.771288 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:41.771289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:41.771450 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:41.771499 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:41.771500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:41.772129 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:41.772130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:41.772777 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:41.772781 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:41.773297 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:41.773298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:41.773360 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:41.773859 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:41.773863 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:41.775798 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":80,"threads_started":1}
W20260926 20:56:41.776063 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:41.771023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39943 (local address 127.0.98.62:36089)
0926 20:56:41.771129 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:41.771131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:41.771142 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:56:41.771192 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:41.771193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:41.771204 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:41.771265 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:41.771570 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:41.772056 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:41.772914 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:41.773184 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:41.773283 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:41.775989 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39943: BlockingRecv error: recv got EOF from 127.0.98.2:39943 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 20:56:41 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:41 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:41.911278 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:41.906322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.3:53639)
0926 20:56:41.906546 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:41.906556 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:41.906636 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:41.906782 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:41.906783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:41.906963 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:41.907010 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:41.907012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:41.907692 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:41.907693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:41.908236 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:41.908238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:41.908701 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:41.908702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:41.908771 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:41.909232 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:41.909237 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:41.911217 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":101,"threads_started":1}
W20260926 20:56:41.911396 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:41.906426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53639 (local address 127.0.98.62:36089)
0926 20:56:41.906502 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:41.906504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:41.906623 (+ 119us) server_negotiation.cc:407] Connection header received
0926 20:56:41.906709 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:41.906709 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:41.906724 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:41.906777 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:41.907091 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:41.907593 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:41.908325 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:41.908625 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:41.908702 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:41.911318 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53639: BlockingRecv error: recv got EOF from 127.0.98.3:53639 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 20:56:42 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:42 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:42.662635 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:42.657652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:56731)
0926 20:56:42.657807 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:42.657821 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:42.657882 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:42.658015 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:42.658016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:42.658240 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:42.658292 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:42.658311 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:42.659028 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:42.659028 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:42.659712 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:42.659714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:42.660067 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:42.660068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:42.660106 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:42.660723 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:42.660728 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:42.662574 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
W20260926 20:56:42.662828 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:42.657711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56731 (local address 127.0.98.62:36089)
0926 20:56:42.657865 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:42.657866 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:42.657875 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:56:42.657926 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:42.657926 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:42.657936 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:42.657995 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:42.658439 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:42.658954 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:42.659792 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:42.659998 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:42.660064 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:42.662698 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56731: BlockingRecv error: recv got EOF from 127.0.98.1:56731 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
Sep 26 20:56:42 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:42 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:42.781453 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:42.776620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:55527)
0926 20:56:42.776785 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:42.776800 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:42.776875 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:42.777023 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:42.777023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:42.777197 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:42.777243 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:42.777245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:42.777901 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:42.777902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:42.778433 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:42.778435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:42.778937 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:42.778938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:42.779010 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:42.779555 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:42.779560 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:42.781384 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
W20260926 20:56:42.781659 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:42.776747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55527 (local address 127.0.98.62:36089)
0926 20:56:42.776825 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:42.776827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:42.776859 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:56:42.776923 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:42.776924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:42.776934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:42.776987 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:42.777308 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:42.777819 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:42.778548 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:42.778792 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:42.778877 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:42.781596 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55527: BlockingRecv error: recv got EOF from 127.0.98.2:55527 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 20:56:42 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:42 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:42.916898 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:42.911856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.3:54705)
0926 20:56:42.912037 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:42.912054 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:42.912142 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:42.912354 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:42.912355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:42.912580 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:42.912634 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:42.912637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:42.913360 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:42.913362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:42.913902 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:42.913904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:42.914316 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:42.914317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:42.914387 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:42.914890 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:42.914895 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:42.916837 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
W20260926 20:56:42.917107 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:42.911999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54705 (local address 127.0.98.62:36089)
0926 20:56:42.912074 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:42.912077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:42.912118 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:56:42.912244 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:42.912245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:42.912266 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:42.912332 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:42.912732 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:42.913251 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:42.914004 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:42.914202 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:42.914311 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:42.917049 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54705: BlockingRecv error: recv got EOF from 127.0.98.3:54705 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 20:56:43 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:43 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:43.667999 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:43.663340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.1:48577)
0926 20:56:43.663490 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:43.663502 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:43.663564 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:43.663710 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:43.663711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:43.663869 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:43.663915 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:43.663917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:43.664635 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:43.664636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:43.665178 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:43.665179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:43.665539 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:43.665540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:43.665590 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:43.666033 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:43.666037 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:43.667900 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
W20260926 20:56:43.668068 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:43.663408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48577 (local address 127.0.98.62:36089)
0926 20:56:43.663548 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:43.663550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:43.663559 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:56:43.663613 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:43.663614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:43.663624 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:43.663693 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:43.663985 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:43.664533 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:43.665272 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:43.665467 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:43.665523 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:43.667999 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48577: BlockingRecv error: recv got EOF from 127.0.98.1:48577 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
Sep 26 20:56:43 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:56:43 dist-test-slave-fl7g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:56:43.786474 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:43.782035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36089 (local address 127.0.98.2:45765)
0926 20:56:43.782172 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:43.782183 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:43.782247 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:43.782377 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:43.782377 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:43.782542 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:43.782593 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:43.782595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:43.783255 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:43.783256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:43.783809 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:43.783810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:43.784155 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:43.784156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:43.784197 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:43.784663 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:43.784667 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:43.786415 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36089: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
W20260926 20:56:43.786597 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:43.782112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45765 (local address 127.0.98.62:36089)
0926 20:56:43.782198 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:43.782199 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:43.782249 (+ 50us) server_negotiation.cc:407] Connection header received
0926 20:56:43.782315 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:43.782316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:43.782326 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:43.782367 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:43.782655 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:43.783145 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:43.783899 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:43.784086 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:43.784144 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:43.786526 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45765: BlockingRecv error: recv got EOF from 127.0.98.2:45765 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:56:43.835158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 9112
I20260926 20:56:43.839879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 9264
I20260926 20:56:43.844381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 9396
I20260926 20:56:43.848948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 9042
2026-09-26T20:56:43Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17505 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T20:56:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:43Z Disabled control of system clock
I20260926 20:56:43.877296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38547
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 20:56:43.959184 9592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:43.959406 9592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:43.959427 9592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:43.960709 9592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:43.960758 9592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:43.960772 9592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:43.960784 9592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:43.960798 9592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:43.962303 9592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38547
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:43.962594 9592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:43.962932 9592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:43.968140 9598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:43.968133 9600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:43.968143 9597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:43.968329 9592 server_base.cc:1034] running on GCE node
I20260926 20:56:43.968700 9592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:43.968987 9592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:43.970166 9592 hybrid_clock.cc:648] HybridClock initialized: now 1790456203970140 us; error 37 us; skew 500 ppm
I20260926 20:56:43.971797 9592 webserver.cc:460] Webserver started at http://127.0.98.62:44507/ using document root <none> and password file <none>
I20260926 20:56:43.972086 9592 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:43.972139 9592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:43.972334 9592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:43.973621 9592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "3f4fc21dbe824e528808b9e23fdeb164"
format_stamp: "Formatted at 2026-09-26 20:56:43 on dist-test-slave-fl7g"
server_key: "e69a4c38b1fd355f1d2bc3f0699770b8"
server_key_iv: "dc15137f933ca00452edc2bf59e13154"
server_key_version: "encryptionkey@0"
I20260926 20:56:43.974043 9592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "3f4fc21dbe824e528808b9e23fdeb164"
format_stamp: "Formatted at 2026-09-26 20:56:43 on dist-test-slave-fl7g"
server_key: "e69a4c38b1fd355f1d2bc3f0699770b8"
server_key_iv: "dc15137f933ca00452edc2bf59e13154"
server_key_version: "encryptionkey@0"
I20260926 20:56:43.975654 9592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:56:43.976516 9606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:43.976728 9592 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:43.976824 9592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "3f4fc21dbe824e528808b9e23fdeb164"
format_stamp: "Formatted at 2026-09-26 20:56:43 on dist-test-slave-fl7g"
server_key: "e69a4c38b1fd355f1d2bc3f0699770b8"
server_key_iv: "dc15137f933ca00452edc2bf59e13154"
server_key_version: "encryptionkey@0"
I20260926 20:56:43.976883 9592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:43.992713 9592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:43.993026 9592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:43.993134 9592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:43.997112 9592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38547
I20260926 20:56:43.997211 9658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38547 every 8 connection(s)
I20260926 20:56:43.997576 9592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:43.998385 9659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:44.001222 9659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164: Bootstrap starting.
I20260926 20:56:44.001899 9659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:44.002091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 9592
I20260926 20:56:44.002188 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:56:44.002264 9659 log.cc:824] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:44.002442 392 external_mini_cluster.cc:1428] Setting key ccb066129bd71f753701e9da43bd5a92
I20260926 20:56:44.003187 9659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164: No bootstrap required, opened a new log
I20260926 20:56:44.005549 9659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4fc21dbe824e528808b9e23fdeb164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38547 } }
I20260926 20:56:44.005728 9659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:44.005776 9659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4fc21dbe824e528808b9e23fdeb164, State: Initialized, Role: FOLLOWER
I20260926 20:56:44.005947 9659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4fc21dbe824e528808b9e23fdeb164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38547 } }
I20260926 20:56:44.006074 9659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:44.006121 9659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:44.006177 9659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:44.006772 9659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4fc21dbe824e528808b9e23fdeb164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38547 } }
I20260926 20:56:44.006896 9659 leader_election.cc:302] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f4fc21dbe824e528808b9e23fdeb164; no voters:
I20260926 20:56:44.007068 9659 leader_election.cc:288] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:44.007128 9664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:44.007318 9664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164 [term 1 LEADER]: Becoming Leader. State: Replica: 3f4fc21dbe824e528808b9e23fdeb164, State: Running, Role: LEADER
I20260926 20:56:44.007371 9659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:44.007491 9664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4fc21dbe824e528808b9e23fdeb164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38547 } }
I20260926 20:56:44.007922 9666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f4fc21dbe824e528808b9e23fdeb164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4fc21dbe824e528808b9e23fdeb164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38547 } } }
I20260926 20:56:44.007911 9664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f4fc21dbe824e528808b9e23fdeb164. Latest consensus state: current_term: 1 leader_uuid: "3f4fc21dbe824e528808b9e23fdeb164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4fc21dbe824e528808b9e23fdeb164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38547 } } }
I20260926 20:56:44.008092 9664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:44.008291 9666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:44.008476 9672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:44.008556 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:44.003007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.0.98.62:38547)
0926 20:56:44.003256 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:44.003266 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:44.003300 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:56:44.003763 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:44.003770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:44.003872 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:44.004047 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:44.004298 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:44.005267 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:44.006257 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:44.006695 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:44.006835 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:44.007604 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:44.007609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:44.007610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:44.007853 (+ 243us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:44.007898 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:44.007915 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:44.007987 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:44.008224 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 20:56:44.008327 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":106,"threads_started":1}
I20260926 20:56:44.009115 9672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:44.010697 9672 catalog_manager.cc:1347] Generated new cluster ID: 1bb79e6f0b3740edbc8233f20ec57cbb
I20260926 20:56:44.010754 9672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:56:44.034952 9672 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:44.035698 9672 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:44.041011 9672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f4fc21dbe824e528808b9e23fdeb164: Generated new TSK 0
I20260926 20:56:44.041263 9672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:56:44.043138 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38547
--builtin_ntp_servers=127.0.98.20:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 20:56:44.157478 9683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:44.157728 9683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:44.157766 9683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:44.158990 9683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:44.159060 9683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:44.159104 9683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:56:44.160590 9683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:44.160871 9683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:44.161180 9683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:44.167343 9689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:44.167351 9688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:44.167373 9691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:44.167613 9683 server_base.cc:1034] running on GCE node
I20260926 20:56:44.167830 9683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:44.168090 9683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:44.169268 9683 hybrid_clock.cc:648] HybridClock initialized: now 1790456204169251 us; error 32 us; skew 500 ppm
I20260926 20:56:44.170754 9683 webserver.cc:460] Webserver started at http://127.0.98.1:33175/ using document root <none> and password file <none>
I20260926 20:56:44.171034 9683 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:44.171083 9683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:44.171233 9683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:44.172465 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "c44ed35ae14e4c458e1ea5a9a23a9b13"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "29d5269c4e43614c6f169bf33b6d13e4"
server_key_iv: "6d463571081773615a6d5f3c2cd0946e"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.172839 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c44ed35ae14e4c458e1ea5a9a23a9b13"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "29d5269c4e43614c6f169bf33b6d13e4"
server_key_iv: "6d463571081773615a6d5f3c2cd0946e"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.174191 9683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:44.174986 9697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:44.175139 9683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:56:44.175239 9683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "c44ed35ae14e4c458e1ea5a9a23a9b13"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "29d5269c4e43614c6f169bf33b6d13e4"
server_key_iv: "6d463571081773615a6d5f3c2cd0946e"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.175315 9683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:44.185880 9683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:44.186219 9683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:44.186362 9683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:44.186611 9683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:44.187011 9683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:44.187072 9683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:44.187115 9683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:44.187143 9683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:44.193058 9683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40763
I20260926 20:56:44.193203 9810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40763 every 8 connection(s)
I20260926 20:56:44.193523 9683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:56:44.198899 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:44.194337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48515 (local address 127.0.98.62:38547)
0926 20:56:44.194386 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:44.194388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:44.194609 (+ 221us) server_negotiation.cc:407] Connection header received
0926 20:56:44.195149 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:44.195150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:44.195160 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:44.195212 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:44.195931 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:44.196540 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:44.197576 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:44.197765 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:44.197824 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:44.198667 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:44.198669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:44.198669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:44.198673 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:44.198688 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:44.198692 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:44.198740 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:44.198814 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 20:56:44.198837 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:56:44.199155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 9683
I20260926 20:56:44.199177 9813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:44.194314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38547 (local address 127.0.98.1:48515)
0926 20:56:44.194494 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:44.194530 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:44.195059 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:44.195272 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:44.195276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:44.195284 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:44.195829 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:44.195833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:44.196644 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:44.196645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:44.197474 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:44.197476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:44.197844 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:44.197845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:44.197965 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:44.198517 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:56:44.198518 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:44.198549 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:56:44.198578 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:44.198743 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:44.198746 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:44.198749 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:56:44.198797 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 20:56:44.198877 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260926 20:56:44.199265 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:56:44.199482 392 external_mini_cluster.cc:1428] Setting key 03ff0cb664694b66453cb1d9114739ce
I20260926 20:56:44.199753 9811 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38547
I20260926 20:56:44.199879 9811 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:44.200073 9811 heartbeater.cc:511] Master 127.0.98.62:38547 requested a full tablet report, sending...
I20260926 20:56:44.200686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38547
--builtin_ntp_servers=127.0.98.20:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 20:56:44.200711 9623 ts_manager.cc:194] Registered new tserver with Master: c44ed35ae14e4c458e1ea5a9a23a9b13 (127.0.98.1:40763)
I20260926 20:56:44.201666 9623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:48515
W20260926 20:56:44.292141 9814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:44.292367 9814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:44.292394 9814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:44.293900 9814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:44.293954 9814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:44.293984 9814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:56:44.295763 9814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:44.296046 9814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:44.296386 9814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:44.301832 9820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:44.301923 9814 server_base.cc:1034] running on GCE node
W20260926 20:56:44.301859 9819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:44.302052 9822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:44.302310 9814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:44.302579 9814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:44.303771 9814 hybrid_clock.cc:648] HybridClock initialized: now 1790456204303762 us; error 30 us; skew 500 ppm
I20260926 20:56:44.305183 9814 webserver.cc:460] Webserver started at http://127.0.98.2:38173/ using document root <none> and password file <none>
I20260926 20:56:44.305450 9814 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:44.305521 9814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:44.305707 9814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:44.306903 9814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "b7bfdfa51fb845aab10560eacbcfe2cc"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "df91ccb0c4a3c58acb1b49e2910bc2ef"
server_key_iv: "c11394cef7f57313b9288c25d03db6de"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.307274 9814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b7bfdfa51fb845aab10560eacbcfe2cc"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "df91ccb0c4a3c58acb1b49e2910bc2ef"
server_key_iv: "c11394cef7f57313b9288c25d03db6de"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.308924 9814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:56:44.309788 9828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:44.310006 9814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:44.310113 9814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "b7bfdfa51fb845aab10560eacbcfe2cc"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "df91ccb0c4a3c58acb1b49e2910bc2ef"
server_key_iv: "c11394cef7f57313b9288c25d03db6de"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.310212 9814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:44.320842 9814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:44.321170 9814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:44.321319 9814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:44.321589 9814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:44.321913 9814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:44.321969 9814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:44.322014 9814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:44.322046 9814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:44.327801 9814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34515
I20260926 20:56:44.327849 9941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34515 every 8 connection(s)
I20260926 20:56:44.328176 9814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:56:44.333355 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:44.328946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55035 (local address 127.0.98.62:38547)
0926 20:56:44.329020 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:44.329022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:44.329275 (+ 253us) server_negotiation.cc:407] Connection header received
0926 20:56:44.329768 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:44.329769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:44.329780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:44.329843 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:44.330477 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:44.330964 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:44.331924 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:44.332116 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:44.332174 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:44.333139 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:44.333141 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:44.333141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:44.333145 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:44.333159 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:44.333163 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:44.333208 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:44.333274 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 20:56:44.333295 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:56:44.333592 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:44.328930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38547 (local address 127.0.98.2:55035)
0926 20:56:44.329135 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:44.329177 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:44.329675 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:44.329895 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:44.329898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:44.329906 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:44.330401 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:44.330405 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:44.331044 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:44.331044 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:44.331834 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:44.331836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:44.332197 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:44.332198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:44.332308 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:44.332997 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:56:44.332998 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:44.333033 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:56:44.333056 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:44.333208 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:44.333210 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:44.333213 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:56:44.333259 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 20:56:44.333338 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260926 20:56:44.334138 9942 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38547
I20260926 20:56:44.334245 9942 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:44.334425 9942 heartbeater.cc:511] Master 127.0.98.62:38547 requested a full tablet report, sending...
I20260926 20:56:44.334848 9623 ts_manager.cc:194] Registered new tserver with Master: b7bfdfa51fb845aab10560eacbcfe2cc (127.0.98.2:34515)
I20260926 20:56:44.335227 9623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:55035
I20260926 20:56:44.336210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 9814
I20260926 20:56:44.336293 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:56:44.336490 392 external_mini_cluster.cc:1428] Setting key f5bbe69aee89efa0e13163c8bb21e8c5
I20260926 20:56:44.337352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38547
--builtin_ntp_servers=127.0.98.20:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 20:56:44.420361 9945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:44.420627 9945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:44.420651 9945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:44.422051 9945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:44.422107 9945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:44.422135 9945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:56:44.423647 9945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:44.423862 9945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:44.424124 9945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:44.429342 9950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:44.429350 9953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:44.429368 9945 server_base.cc:1034] running on GCE node
W20260926 20:56:44.429350 9951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:44.429816 9945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:44.430087 9945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:44.431241 9945 hybrid_clock.cc:648] HybridClock initialized: now 1790456204431227 us; error 28 us; skew 500 ppm
I20260926 20:56:44.432693 9945 webserver.cc:460] Webserver started at http://127.0.98.3:36777/ using document root <none> and password file <none>
I20260926 20:56:44.432936 9945 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:44.432986 9945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:44.433081 9945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:44.434126 9945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "d8c4acd0b8d148b1906775a9d3811b0c"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "181e118f8443fbca1754447e2ecfb64c"
server_key_iv: "974b1a0de944b8765018a5ee25bb720f"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.434449 9945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d8c4acd0b8d148b1906775a9d3811b0c"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "181e118f8443fbca1754447e2ecfb64c"
server_key_iv: "974b1a0de944b8765018a5ee25bb720f"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.435783 9945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:56:44.436659 9959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:44.436839 9945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:44.436925 9945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "d8c4acd0b8d148b1906775a9d3811b0c"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "181e118f8443fbca1754447e2ecfb64c"
server_key_iv: "974b1a0de944b8765018a5ee25bb720f"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.436983 9945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:44.446033 9945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:44.446293 9945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:44.446396 9945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:44.446578 9945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:44.447032 9945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:44.447069 9945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:44.447091 9945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:44.447105 9945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:44.452749 9945 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35999
I20260926 20:56:44.452824 10072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35999 every 8 connection(s)
I20260926 20:56:44.453147 9945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:56:44.458448 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:44.453956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43825 (local address 127.0.98.62:38547)
0926 20:56:44.454040 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:44.454042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:44.454211 (+ 169us) server_negotiation.cc:407] Connection header received
0926 20:56:44.454745 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:44.454746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:44.454756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:44.454813 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:44.455446 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:44.455924 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:44.457037 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:44.457223 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:44.457283 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:44.458211 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:44.458212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:44.458213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:44.458217 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:44.458231 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:44.458235 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:44.458281 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:44.458368 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 20:56:44.458388 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:56:44.458668 10075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:44.453904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38547 (local address 127.0.98.3:43825)
0926 20:56:44.454089 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:44.454124 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:44.454650 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:44.454872 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:44.454876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:44.454884 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:44.455374 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:44.455379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:44.456013 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:44.456014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:44.456949 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:44.456952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:44.457305 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:44.457306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:44.457421 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:44.458072 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:56:44.458073 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:44.458102 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:56:44.458126 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:44.458297 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:44.458301 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:44.458303 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:56:44.458361 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:56:44.458441 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260926 20:56:44.459239 10073 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38547
I20260926 20:56:44.459348 10073 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:44.459550 10073 heartbeater.cc:511] Master 127.0.98.62:38547 requested a full tablet report, sending...
I20260926 20:56:44.459945 9623 ts_manager.cc:194] Registered new tserver with Master: d8c4acd0b8d148b1906775a9d3811b0c (127.0.98.3:35999)
I20260926 20:56:44.460279 9623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:43825
I20260926 20:56:44.462280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 9945
I20260926 20:56:44.462404 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:56:44.462615 392 external_mini_cluster.cc:1428] Setting key 32343ba5ae69d1e03d7e6e5404e59c66
I20260926 20:56:44.463785 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:56:44.465823 9663 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:44.465216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59660 (local address 127.0.98.62:38547)
0926 20:56:44.465294 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:44.465296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:44.465409 (+ 113us) server_negotiation.cc:407] Connection header received
0926 20:56:44.465507 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:44.465508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:44.465519 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:44.465592 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:44.465751 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59660: BlockingRecv error: recv got EOF from 127.0.0.1:59660 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:56:44.465858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 9683
I20260926 20:56:44.470980 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 9814
I20260926 20:56:44.475445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 9945
I20260926 20:56:44.479882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 9592
2026-09-26T20:56:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (629 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T20:56:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:44Z Disabled control of system clock
I20260926 20:56:44.507038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39601
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 20:56:44.588037 10084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:44.588263 10084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:44.588286 10084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:44.589496 10084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:44.589548 10084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:44.589561 10084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:44.589573 10084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:44.589586 10084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:44.590902 10084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:44.591173 10084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:44.591429 10084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:56:44.596591 10084 server_base.cc:1034] running on GCE node
W20260926 20:56:44.596511 10092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:44.596555 10090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:44.596511 10089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:44.597060 10084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:44.597393 10084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:44.598580 10084 hybrid_clock.cc:648] HybridClock initialized: now 1790456204598557 us; error 40 us; skew 500 ppm
I20260926 20:56:44.599961 10084 webserver.cc:460] Webserver started at http://127.0.98.62:41825/ using document root <none> and password file <none>
I20260926 20:56:44.600220 10084 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:44.600270 10084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:44.600397 10084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:44.601603 10084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "e8cefbd398194027b112e86cf65349b8"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "bf43517df6b8a903e8ab440a3f13a051"
server_key_iv: "51f12eb4bdff00769c72d301b115e78b"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.601969 10084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "e8cefbd398194027b112e86cf65349b8"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "bf43517df6b8a903e8ab440a3f13a051"
server_key_iv: "51f12eb4bdff00769c72d301b115e78b"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.603361 10084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:56:44.604195 10098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:44.604389 10084 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:44.604508 10084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "e8cefbd398194027b112e86cf65349b8"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "bf43517df6b8a903e8ab440a3f13a051"
server_key_iv: "51f12eb4bdff00769c72d301b115e78b"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.604573 10084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:44.608151 10084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:44.608510 10084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:44.608660 10084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:44.612335 10084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39601
I20260926 20:56:44.612368 10150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39601 every 8 connection(s)
I20260926 20:56:44.612748 10084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:44.613629 10151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:44.616497 10151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8: Bootstrap starting.
I20260926 20:56:44.617074 10151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:44.617316 10151 log.cc:824] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:44.618006 10151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8: No bootstrap required, opened a new log
I20260926 20:56:44.619581 10151 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8cefbd398194027b112e86cf65349b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } }
I20260926 20:56:44.619729 10151 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:44.619752 10151 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8cefbd398194027b112e86cf65349b8, State: Initialized, Role: FOLLOWER
I20260926 20:56:44.619903 10151 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8cefbd398194027b112e86cf65349b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } }
I20260926 20:56:44.619984 10151 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:44.620009 10151 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:44.620045 10151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:44.620692 10151 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8cefbd398194027b112e86cf65349b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } }
I20260926 20:56:44.620785 10151 leader_election.cc:302] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8cefbd398194027b112e86cf65349b8; no voters:
I20260926 20:56:44.620925 10151 leader_election.cc:288] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:44.621064 10154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:44.621235 10154 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8 [term 1 LEADER]: Becoming Leader. State: Replica: e8cefbd398194027b112e86cf65349b8, State: Running, Role: LEADER
I20260926 20:56:44.621241 10151 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:44.621356 10154 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8cefbd398194027b112e86cf65349b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } }
I20260926 20:56:44.621601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 10084
I20260926 20:56:44.621692 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:56:44.621933 392 external_mini_cluster.cc:1428] Setting key 95697b57dc928329c2816e2015398a7b
I20260926 20:56:44.622116 10156 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8cefbd398194027b112e86cf65349b8. Latest consensus state: current_term: 1 leader_uuid: "e8cefbd398194027b112e86cf65349b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8cefbd398194027b112e86cf65349b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } } }
I20260926 20:56:44.622195 10156 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:44.622501 10155 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8cefbd398194027b112e86cf65349b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8cefbd398194027b112e86cf65349b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } } }
I20260926 20:56:44.622563 10155 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:44.622655 10163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:44.623237 10163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:44.624986 10163 catalog_manager.cc:1347] Generated new cluster ID: e8567085d63647ffb88f24535d26316d
I20260926 20:56:44.625048 10163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:56:44.628623 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:44.622699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52980 (local address 127.0.98.62:39601)
0926 20:56:44.623191 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:44.623199 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:44.623223 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:56:44.623358 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:44.623366 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:44.623411 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:44.623566 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:44.624497 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:44.624506 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:44.624507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:44.624511 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:44.624592 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:44.624608 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:44.624675 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:44.624821 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 20:56:44.624936 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":76,"threads_started":1}
I20260926 20:56:44.634857 10163 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:44.635031 10163 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:44.647002 10163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8cefbd398194027b112e86cf65349b8: Generated new TSK 0
I20260926 20:56:44.647243 10163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:56:44.652750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--builtin_ntp_servers=127.0.98.20:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 20:56:44.736269 10175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:44.736568 10175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:44.736598 10175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:44.737952 10175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:44.737996 10175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:44.738024 10175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:56:44.739429 10175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:44.739707 10175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:44.740036 10175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:44.745234 10183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:44.745276 10180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:44.745290 10181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:44.745388 10175 server_base.cc:1034] running on GCE node
I20260926 20:56:44.745685 10175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:44.745985 10175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:44.747165 10175 hybrid_clock.cc:648] HybridClock initialized: now 1790456204747119 us; error 60 us; skew 500 ppm
I20260926 20:56:44.748845 10175 webserver.cc:460] Webserver started at http://127.0.98.1:39117/ using document root <none> and password file <none>
I20260926 20:56:44.749142 10175 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:44.749212 10175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:44.749356 10175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:44.750754 10175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "4b3d6f96d6ea4a8f905a44769d4ed736"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "7c1aeb9ba77572bcdeae8e19f83f4c13"
server_key_iv: "28bab3406506ce98629f1cc0557f0d44"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.751138 10175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4b3d6f96d6ea4a8f905a44769d4ed736"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "7c1aeb9ba77572bcdeae8e19f83f4c13"
server_key_iv: "28bab3406506ce98629f1cc0557f0d44"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.752758 10175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260926 20:56:44.753594 10189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:44.753788 10175 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:44.753876 10175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "4b3d6f96d6ea4a8f905a44769d4ed736"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "7c1aeb9ba77572bcdeae8e19f83f4c13"
server_key_iv: "28bab3406506ce98629f1cc0557f0d44"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.753947 10175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:44.759009 10175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:44.759342 10175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:44.759490 10175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:44.759757 10175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:44.760087 10175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:44.760119 10175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:44.760141 10175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:44.760154 10175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:44.766012 10175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38533
I20260926 20:56:44.766115 10302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38533 every 8 connection(s)
I20260926 20:56:44.766417 10175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:56:44.767710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 10175
I20260926 20:56:44.767815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:56:44.768007 392 external_mini_cluster.cc:1428] Setting key 5630c1b18d5f5896f484a433d2156639
I20260926 20:56:44.769048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--builtin_ntp_servers=127.0.98.20:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 20:56:44.770020 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:44.767192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35487 (local address 127.0.98.62:39601)
0926 20:56:44.767251 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:44.767253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:44.767480 (+ 227us) server_negotiation.cc:407] Connection header received
0926 20:56:44.768231 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:44.768232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:44.768240 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:44.768312 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:44.769775 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:44.769777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:44.769777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:44.769782 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:44.769798 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:44.769803 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:44.769851 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:44.769935 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 20:56:44.769949 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 20:56:44.770243 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:44.767143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39601 (local address 127.0.98.1:35487)
0926 20:56:44.767345 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:44.767388 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:44.768093 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:44.768497 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:44.768502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:44.768512 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:44.769608 (+ 1096us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:56:44.769608 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:44.769639 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:56:44.769677 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:44.769858 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:44.769863 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:44.769866 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:56:44.769945 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:56:44.770011 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
I20260926 20:56:44.770772 10303 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39601
I20260926 20:56:44.770881 10303 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:44.771054 10303 heartbeater.cc:511] Master 127.0.98.62:39601 requested a full tablet report, sending...
I20260926 20:56:44.771610 10115 ts_manager.cc:194] Registered new tserver with Master: 4b3d6f96d6ea4a8f905a44769d4ed736 (127.0.98.1:38533)
W20260926 20:56:44.852491 10306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:44.852721 10306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:44.852746 10306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:44.853911 10306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:44.853955 10306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:44.853981 10306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:56:44.855361 10306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:44.855574 10306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:44.855856 10306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:44.861577 10311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:44.861577 10314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:44.861615 10312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:44.862017 10306 server_base.cc:1034] running on GCE node
I20260926 20:56:44.862254 10306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:44.862510 10306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:44.863680 10306 hybrid_clock.cc:648] HybridClock initialized: now 1790456204863654 us; error 38 us; skew 500 ppm
I20260926 20:56:44.865170 10306 webserver.cc:460] Webserver started at http://127.0.98.2:43203/ using document root <none> and password file <none>
I20260926 20:56:44.865449 10306 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:44.865515 10306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:44.865697 10306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:44.866899 10306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "e77269657ed1444aab7498436e087b03"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "4e62b8b3cbe579b0419e16d2a65146c9"
server_key_iv: "fca8df650597e3528ddbfb6f2c8e1eab"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.867347 10306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e77269657ed1444aab7498436e087b03"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "4e62b8b3cbe579b0419e16d2a65146c9"
server_key_iv: "fca8df650597e3528ddbfb6f2c8e1eab"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.869020 10306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 20:56:44.869959 10320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:44.870178 10306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:44.870296 10306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "e77269657ed1444aab7498436e087b03"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "4e62b8b3cbe579b0419e16d2a65146c9"
server_key_iv: "fca8df650597e3528ddbfb6f2c8e1eab"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.870388 10306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:44.874177 10306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:44.874509 10306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:44.874676 10306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:44.874912 10306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:44.875268 10306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:44.875327 10306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:44.875375 10306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:44.875411 10306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:44.882601 10306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36837
I20260926 20:56:44.882652 10433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36837 every 8 connection(s)
I20260926 20:56:44.883038 10306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:56:44.884248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 10306
I20260926 20:56:44.884371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:56:44.884584 392 external_mini_cluster.cc:1428] Setting key 64489299e1cf539a6bb43cf88c7b6ce3
I20260926 20:56:44.885710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--builtin_ntp_servers=127.0.98.20:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 20:56:44.886665 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:44.883898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43075 (local address 127.0.98.62:39601)
0926 20:56:44.884027 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:44.884030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:44.884239 (+ 209us) server_negotiation.cc:407] Connection header received
0926 20:56:44.884932 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:44.884933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:44.884938 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:44.884984 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:44.886403 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:44.886404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:44.886404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:44.886408 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:44.886425 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:44.886430 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:44.886480 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:44.886591 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 20:56:44.886604 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:56:44.886916 10436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:44.883860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39601 (local address 127.0.98.2:43075)
0926 20:56:44.884099 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:44.884147 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:44.884796 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:44.885068 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:44.885072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:44.885083 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:44.886221 (+ 1138us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:56:44.886221 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:44.886266 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:56:44.886306 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:44.886509 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:44.886515 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:44.886518 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:56:44.886597 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:56:44.886664 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
I20260926 20:56:44.887435 10434 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39601
I20260926 20:56:44.887544 10434 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:44.887708 10434 heartbeater.cc:511] Master 127.0.98.62:39601 requested a full tablet report, sending...
I20260926 20:56:44.888068 10115 ts_manager.cc:194] Registered new tserver with Master: e77269657ed1444aab7498436e087b03 (127.0.98.2:36837)
W20260926 20:56:44.971417 10437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:44.971630 10437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:44.971653 10437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:44.972886 10437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:44.972939 10437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:44.972966 10437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:56:44.974560 10437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:44.974884 10437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:44.975206 10437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:44.980620 10443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:44.980628 10442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:44.980700 10437 server_base.cc:1034] running on GCE node
W20260926 20:56:44.980643 10445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:44.981040 10437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:44.981307 10437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:44.982520 10437 hybrid_clock.cc:648] HybridClock initialized: now 1790456204982491 us; error 63 us; skew 500 ppm
I20260926 20:56:44.984102 10437 webserver.cc:460] Webserver started at http://127.0.98.3:41911/ using document root <none> and password file <none>
I20260926 20:56:44.984373 10437 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:44.984483 10437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:44.984644 10437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:44.985767 10437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "a268a69455a84a77a1d4e7514ae53383"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "7b71d855be47373a1e9b3c4b06621a3a"
server_key_iv: "896a30077c9d1d20f4761145bb474bfe"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.986172 10437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a268a69455a84a77a1d4e7514ae53383"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "7b71d855be47373a1e9b3c4b06621a3a"
server_key_iv: "896a30077c9d1d20f4761145bb474bfe"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.987592 10437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:56:44.988377 10451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:44.988600 10437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:44.988703 10437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "a268a69455a84a77a1d4e7514ae53383"
format_stamp: "Formatted at 2026-09-26 20:56:44 on dist-test-slave-fl7g"
server_key: "7b71d855be47373a1e9b3c4b06621a3a"
server_key_iv: "896a30077c9d1d20f4761145bb474bfe"
server_key_version: "encryptionkey@0"
I20260926 20:56:44.988793 10437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:44.992362 10437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:44.992704 10437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:44.992865 10437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:44.993115 10437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:44.993459 10437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:44.993525 10437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:44.993569 10437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:44.993599 10437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:45.000664 10437 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33933
I20260926 20:56:45.000699 10564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33933 every 8 connection(s)
I20260926 20:56:45.001070 10437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:56:45.001199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 10437
I20260926 20:56:45.001276 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:56:45.001526 392 external_mini_cluster.cc:1428] Setting key 515bf27f946d1d1034b116612c483010
I20260926 20:56:45.004251 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:45.001900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51111 (local address 127.0.98.62:39601)
0926 20:56:45.001971 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:45.001973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:45.002266 (+ 293us) server_negotiation.cc:407] Connection header received
0926 20:56:45.002741 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:45.002741 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:45.002747 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:45.002794 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:45.004055 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:45.004056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:45.004057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:45.004060 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:45.004072 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:45.004076 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:45.004120 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:45.004183 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 20:56:45.004196 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:56:45.004492 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:45.001926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39601 (local address 127.0.98.3:51111)
0926 20:56:45.002129 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:45.002176 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:45.002657 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:45.002848 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:45.002851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:45.002860 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:45.003879 (+ 1019us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:56:45.003879 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:45.003910 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:56:45.003944 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:45.004120 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:45.004123 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:45.004125 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:56:45.004189 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 20:56:45.004237 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260926 20:56:45.005064 10565 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39601
I20260926 20:56:45.005152 10565 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:45.005272 10565 heartbeater.cc:511] Master 127.0.98.62:39601 requested a full tablet report, sending...
I20260926 20:56:45.005724 10115 ts_manager.cc:194] Registered new tserver with Master: a268a69455a84a77a1d4e7514ae53383 (127.0.98.3:33933)
I20260926 20:56:45.013244 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:56:45.015295 10172 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:56:45.014738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.0.98.62:39601)
0926 20:56:45.014782 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:45.014784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:45.014920 (+ 136us) server_negotiation.cc:407] Connection header received
0926 20:56:45.014980 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:45.014981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:45.014985 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:45.015040 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:45.015212 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52994: BlockingRecv error: recv got EOF from 127.0.0.1:52994 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:56:45.015359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 10175
I20260926 20:56:45.020808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 10306
I20260926 20:56:45.024875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 10437
I20260926 20:56:45.029042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 10084
2026-09-26T20:56:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (548 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:56:45 dist-test-slave-fl7g krb5kdc[10575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:56:45 dist-test-slave-fl7g krb5kdc[10575](info): set up 2 sockets
Sep 26 20:56:45 dist-test-slave-fl7g krb5kdc[10575](info): commencing operation
krb5kdc: starting...
W20260926 20:56:46.081565 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:56:46 dist-test-slave-fl7g krb5kdc[10575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:56:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:56:46.208510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46859
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:46.291802 10588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:46.292091 10588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:46.292133 10588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:46.293488 10588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:46.293569 10588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:46.293592 10588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:46.293613 10588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:46.293648 10588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:46.295248 10588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:46.295544 10588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:46.295786 10588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:46.301018 10593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:46.300999 10594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:46.301002 10596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:46.301084 10588 server_base.cc:1034] running on GCE node
I20260926 20:56:46.301455 10588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:46.301766 10588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:46.302950 10588 hybrid_clock.cc:648] HybridClock initialized: now 1790456206302912 us; error 50 us; skew 500 ppm
Sep 26 20:56:46 dist-test-slave-fl7g krb5kdc[10575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:46.305922 10588 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:56:46.306445 10588 webserver.cc:460] Webserver started at http://127.0.98.62:44343/ using document root <none> and password file <none>
I20260926 20:56:46.306689 10588 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:46.306735 10588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:46.306882 10588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:46.307964 10588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "cd023341ead3459e83a74a7b20b75e63"
format_stamp: "Formatted at 2026-09-26 20:56:46 on dist-test-slave-fl7g"
server_key: "982e3f7a90f5ce0da0f4affb68ec0d09"
server_key_iv: "424f4a216add1f07cbfa2b8814762aad"
server_key_version: "encryptionkey@0"
I20260926 20:56:46.308343 10588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "cd023341ead3459e83a74a7b20b75e63"
format_stamp: "Formatted at 2026-09-26 20:56:46 on dist-test-slave-fl7g"
server_key: "982e3f7a90f5ce0da0f4affb68ec0d09"
server_key_iv: "424f4a216add1f07cbfa2b8814762aad"
server_key_version: "encryptionkey@0"
I20260926 20:56:46.309824 10588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:46.310683 10603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:46.310892 10588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:56:46.310997 10588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "cd023341ead3459e83a74a7b20b75e63"
format_stamp: "Formatted at 2026-09-26 20:56:46 on dist-test-slave-fl7g"
server_key: "982e3f7a90f5ce0da0f4affb68ec0d09"
server_key_iv: "424f4a216add1f07cbfa2b8814762aad"
server_key_version: "encryptionkey@0"
I20260926 20:56:46.311085 10588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:46.320659 10588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:46.323032 10588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:46.323200 10588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:46.327414 10588 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46859
I20260926 20:56:46.327447 10655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46859 every 8 connection(s)
I20260926 20:56:46.327832 10588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:46.329027 10656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:46.331503 10656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63: Bootstrap starting.
I20260926 20:56:46.332150 10656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:46.332506 10656 log.cc:824] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:46.333613 10656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63: No bootstrap required, opened a new log
I20260926 20:56:46.333691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 10588
I20260926 20:56:46.333782 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:56:46.333994 392 external_mini_cluster.cc:1428] Setting key b2041550badfe4278ade85d142c62723
I20260926 20:56:46.335723 10656 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd023341ead3459e83a74a7b20b75e63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } }
I20260926 20:56:46.335907 10656 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:46.335950 10656 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd023341ead3459e83a74a7b20b75e63, State: Initialized, Role: FOLLOWER
I20260926 20:56:46.336134 10656 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd023341ead3459e83a74a7b20b75e63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } }
I20260926 20:56:46.336252 10656 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:46.336290 10656 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:46.336344 10656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:46.337441 10656 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd023341ead3459e83a74a7b20b75e63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } }
I20260926 20:56:46.337546 10656 leader_election.cc:302] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd023341ead3459e83a74a7b20b75e63; no voters:
I20260926 20:56:46.337733 10656 leader_election.cc:288] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:46.337783 10661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:46.337978 10661 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63 [term 1 LEADER]: Becoming Leader. State: Replica: cd023341ead3459e83a74a7b20b75e63, State: Running, Role: LEADER
I20260926 20:56:46.338042 10656 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:46.338164 10661 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd023341ead3459e83a74a7b20b75e63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } }
I20260926 20:56:46.339236 10662 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd023341ead3459e83a74a7b20b75e63. Latest consensus state: current_term: 1 leader_uuid: "cd023341ead3459e83a74a7b20b75e63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd023341ead3459e83a74a7b20b75e63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } } }
I20260926 20:56:46.339351 10662 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:46.339781 10663 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd023341ead3459e83a74a7b20b75e63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd023341ead3459e83a74a7b20b75e63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46859 } } }
I20260926 20:56:46.339859 10663 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:46.340049 10676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:46.340618 10676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 20:56:46 dist-test-slave-fl7g krb5kdc[10575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:46.342291 10676 catalog_manager.cc:1347] Generated new cluster ID: c42b70a887734398af24588be91f4e17
I20260926 20:56:46.342342 10676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:56:46.346343 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:46.334575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59736 (local address 127.0.98.62:46859)
0926 20:56:46.334807 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:46.334817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:46.334852 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:56:46.335696 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:46.335703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:46.335754 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:46.335879 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:46.336761 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.337802 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.338778 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.339424 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.339521 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:46.342542 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:46.342547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:46.342547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:46.342560 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:46.344646 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:46.345064 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:46.345067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:46.345068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:46.345195 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:46.345414 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:46.345415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:46.345415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:46.345718 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:46.345862 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:46.346013 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 20:56:46.346145 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":112,"threads_started":1}
I20260926 20:56:46.355602 10676 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:46.356153 10676 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:46.366896 10676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63: Generated new TSK 0
I20260926 20:56:46.367146 10676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:56:46.403625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46859
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:46.492573 10682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:46.492835 10682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:46.492862 10682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:46.494015 10682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:46.494050 10682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:46.494077 10682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:56:46.495508 10682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:46.495711 10682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:46.495901 10682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:46.501127 10687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:46.501185 10688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:46.501151 10690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:46.501398 10682 server_base.cc:1034] running on GCE node
I20260926 20:56:46.501617 10682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:46.501900 10682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:46.503088 10682 hybrid_clock.cc:648] HybridClock initialized: now 1790456206503091 us; error 49 us; skew 500 ppm
Sep 26 20:56:46 dist-test-slave-fl7g krb5kdc[10575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:46.505839 10682 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:56:46.506369 10682 webserver.cc:460] Webserver started at http://127.0.98.1:36443/ using document root <none> and password file <none>
I20260926 20:56:46.506630 10682 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:46.506703 10682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:46.506855 10682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:46.507922 10682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "ab5d8a2b1a464fe4b6a977c9822360ad"
format_stamp: "Formatted at 2026-09-26 20:56:46 on dist-test-slave-fl7g"
server_key: "500025cabdcccc2d97f0c5ee20382242"
server_key_iv: "a791e17e53b5797a91f35c96e0ec7262"
server_key_version: "encryptionkey@0"
I20260926 20:56:46.508296 10682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ab5d8a2b1a464fe4b6a977c9822360ad"
format_stamp: "Formatted at 2026-09-26 20:56:46 on dist-test-slave-fl7g"
server_key: "500025cabdcccc2d97f0c5ee20382242"
server_key_iv: "a791e17e53b5797a91f35c96e0ec7262"
server_key_version: "encryptionkey@0"
I20260926 20:56:46.509703 10682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:46.510524 10697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:46.510722 10682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:46.510815 10682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "ab5d8a2b1a464fe4b6a977c9822360ad"
format_stamp: "Formatted at 2026-09-26 20:56:46 on dist-test-slave-fl7g"
server_key: "500025cabdcccc2d97f0c5ee20382242"
server_key_iv: "a791e17e53b5797a91f35c96e0ec7262"
server_key_version: "encryptionkey@0"
I20260926 20:56:46.510897 10682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:46.518980 10682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:46.520956 10682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:46.521126 10682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:46.521358 10682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:46.521669 10682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:46.521720 10682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:46.521761 10682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:46.521790 10682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:46.527568 10682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41889
I20260926 20:56:46.527606 10810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41889 every 8 connection(s)
I20260926 20:56:46.527961 10682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:56:46.529518 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 10682
I20260926 20:56:46.529621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:56:46.529848 392 external_mini_cluster.cc:1428] Setting key 7a2a0fe097e6e607bddaefc40a120868
Sep 26 20:56:46 dist-test-slave-fl7g krb5kdc[10575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 20:56:46.539126 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:46.528771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52647 (local address 127.0.98.62:46859)
0926 20:56:46.528850 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:46.528852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:46.529083 (+ 231us) server_negotiation.cc:407] Connection header received
0926 20:56:46.529740 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:46.529741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:46.529755 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:46.529800 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:46.531157 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.531652 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.532905 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.533146 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.533261 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:46.535748 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:46.535749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:46.535749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:46.535760 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:46.537361 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:46.537805 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:46.537807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:46.537807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:46.537870 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:46.538281 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:46.538283 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:46.538283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:46.538414 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:46.538481 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:46.538795 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 20:56:46.538970 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:56:46.539175 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:46.528754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.1:52647)
0926 20:56:46.528940 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:46.528984 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:46.529610 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:46.529879 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:46.529884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:46.530753 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:46.531039 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:46.531047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.531770 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.531771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:46.532691 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:46.532695 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.533271 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.533273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:46.533411 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:46.534013 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:46.534021 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:46.535643 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:46.537459 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:46.537466 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:46.537468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:46.537709 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:46.537979 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:46.537980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:46.537981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:46.538173 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:46.538488 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:46.538493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:46.538619 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 20:56:46.538735 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 20:56:46.538867 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":113,"threads_started":1}
I20260926 20:56:46.539691 10811 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46859
I20260926 20:56:46.539789 10811 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:46.540012 10811 heartbeater.cc:511] Master 127.0.98.62:46859 requested a full tablet report, sending...
I20260926 20:56:46.540686 10620 ts_manager.cc:194] Registered new tserver with Master: ab5d8a2b1a464fe4b6a977c9822360ad (127.0.98.1:41889)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 20:56:46.541468 10620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:56:46.555858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46859
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:46.638761 10816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:46.639017 10816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:46.639057 10816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:46.640316 10816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:46.640393 10816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:46.640465 10816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:56:46.641963 10816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:46.642253 10816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:46.642503 10816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:46.647610 10824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:46.647610 10822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:46.647661 10821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:46.647681 10816 server_base.cc:1034] running on GCE node
I20260926 20:56:46.648026 10816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:46.648237 10816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:46.649384 10816 hybrid_clock.cc:648] HybridClock initialized: now 1790456206649361 us; error 36 us; skew 500 ppm
Sep 26 20:56:46 dist-test-slave-fl7g krb5kdc[10575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:46.651978 10816 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:56:46.652463 10816 webserver.cc:460] Webserver started at http://127.0.98.2:41041/ using document root <none> and password file <none>
I20260926 20:56:46.652704 10816 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:46.652776 10816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:46.652923 10816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:46.654004 10816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "ae4db3ff874644b19dc83688ee622da7"
format_stamp: "Formatted at 2026-09-26 20:56:46 on dist-test-slave-fl7g"
server_key: "fe33093a4271a2c24d0d6a298d11aa9f"
server_key_iv: "e486b288ddc819240c736f9dea593308"
server_key_version: "encryptionkey@0"
I20260926 20:56:46.654382 10816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ae4db3ff874644b19dc83688ee622da7"
format_stamp: "Formatted at 2026-09-26 20:56:46 on dist-test-slave-fl7g"
server_key: "fe33093a4271a2c24d0d6a298d11aa9f"
server_key_iv: "e486b288ddc819240c736f9dea593308"
server_key_version: "encryptionkey@0"
I20260926 20:56:46.655817 10816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:56:46.656736 10831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:46.656937 10816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:56:46.657038 10816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "ae4db3ff874644b19dc83688ee622da7"
format_stamp: "Formatted at 2026-09-26 20:56:46 on dist-test-slave-fl7g"
server_key: "fe33093a4271a2c24d0d6a298d11aa9f"
server_key_iv: "e486b288ddc819240c736f9dea593308"
server_key_version: "encryptionkey@0"
I20260926 20:56:46.657123 10816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:46.672371 10816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:46.674672 10816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:46.674853 10816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:46.675132 10816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:46.675495 10816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:46.675559 10816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:46.675604 10816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:46.675637 10816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:46.682129 10816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43521
I20260926 20:56:46.682209 10944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43521 every 8 connection(s)
I20260926 20:56:46.682582 10816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:56:46 dist-test-slave-fl7g krb5kdc[10575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:46.691229 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 10816
I20260926 20:56:46.691349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:56:46.691596 392 external_mini_cluster.cc:1428] Setting key d4192310685b88e867274003a73b80b5
I20260926 20:56:46.692790 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:46.683440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35509 (local address 127.0.98.62:46859)
0926 20:56:46.683515 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:46.683517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:46.683643 (+ 126us) server_negotiation.cc:407] Connection header received
0926 20:56:46.684141 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:46.684142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:46.684153 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:46.684218 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:46.685251 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.685769 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.686804 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.687007 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.687073 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:46.689449 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:46.689451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:46.689451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:46.689459 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:46.690785 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:46.691328 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:46.691330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:46.691330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:46.691402 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:46.691725 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:46.691727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:46.691727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:46.691927 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:46.691990 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:46.692646 (+ 656us) server_negotiation.cc:299] Negotiation successful
0926 20:56:46.692725 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:56:46.693059 10947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:46.683342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.2:35509)
0926 20:56:46.683511 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:46.683554 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:46.684036 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:46.684297 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:46.684300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:46.684941 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:46.685140 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:46.685145 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.685874 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.685874 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:46.686693 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:46.686696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.687103 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.687104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:46.687217 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:46.687801 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:46.687809 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:46.689350 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:46.690869 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:46.690875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:46.690877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:46.691208 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:46.691470 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:46.691471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:46.691472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:46.691620 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:46.692215 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:46.692221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:46.692413 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 20:56:46.692558 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 20:56:46.692771 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260926 20:56:46.693629 10945 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46859
I20260926 20:56:46.693747 10945 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:46.693965 10945 heartbeater.cc:511] Master 127.0.98.62:46859 requested a full tablet report, sending...
I20260926 20:56:46.694355 10620 ts_manager.cc:194] Registered new tserver with Master: ae4db3ff874644b19dc83688ee622da7 (127.0.98.2:43521)
I20260926 20:56:46.694867 10620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35509
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:56:46.720175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46859
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:46.801652 10950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:46.801928 10950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:46.801970 10950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:46.803170 10950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:46.803253 10950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:46.803303 10950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:56:46.804891 10950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:56:46.805173 10950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:46.805421 10950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:46.810686 10958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:46.810801 10955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:46.810681 10956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:46.811000 10950 server_base.cc:1034] running on GCE node
I20260926 20:56:46.811264 10950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:46.811528 10950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:46.812695 10950 hybrid_clock.cc:648] HybridClock initialized: now 1790456206812689 us; error 53 us; skew 500 ppm
Sep 26 20:56:46 dist-test-slave-fl7g krb5kdc[10575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:46.815383 10950 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:56:46.815886 10950 webserver.cc:460] Webserver started at http://127.0.98.3:45707/ using document root <none> and password file <none>
I20260926 20:56:46.816128 10950 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:46.816211 10950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:46.816359 10950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:46.817461 10950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "bdbd058609f14cf7ae5667d3543c2557"
format_stamp: "Formatted at 2026-09-26 20:56:46 on dist-test-slave-fl7g"
server_key: "eb2009fe2387ffebae1202a20b4017ea"
server_key_iv: "9813604cbaed15d896903d992dce44f8"
server_key_version: "encryptionkey@0"
I20260926 20:56:46.817842 10950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bdbd058609f14cf7ae5667d3543c2557"
format_stamp: "Formatted at 2026-09-26 20:56:46 on dist-test-slave-fl7g"
server_key: "eb2009fe2387ffebae1202a20b4017ea"
server_key_iv: "9813604cbaed15d896903d992dce44f8"
server_key_version: "encryptionkey@0"
I20260926 20:56:46.819172 10950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:56:46.819883 10965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:46.820055 10950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:56:46.820156 10950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "bdbd058609f14cf7ae5667d3543c2557"
format_stamp: "Formatted at 2026-09-26 20:56:46 on dist-test-slave-fl7g"
server_key: "eb2009fe2387ffebae1202a20b4017ea"
server_key_iv: "9813604cbaed15d896903d992dce44f8"
server_key_version: "encryptionkey@0"
I20260926 20:56:46.820236 10950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:46.864599 10950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:46.867192 10950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:46.867383 10950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:46.867638 10950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:46.867974 10950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:46.868010 10950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:46.868032 10950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:46.868045 10950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:46.874399 10950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39081
I20260926 20:56:46.874435 11078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39081 every 8 connection(s)
I20260926 20:56:46.874794 10950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:56:46.875566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 10950
I20260926 20:56:46.875680 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:56:46.875901 392 external_mini_cluster.cc:1428] Setting key c10a23d409add5c184382888216a3dc0
Sep 26 20:56:46 dist-test-slave-fl7g krb5kdc[10575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:56:46.885430 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:46.875579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52693 (local address 127.0.98.62:46859)
0926 20:56:46.876078 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:46.876080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:46.876310 (+ 230us) server_negotiation.cc:407] Connection header received
0926 20:56:46.877078 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:46.877079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:46.877091 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:46.877135 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:46.878152 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.878653 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.879724 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.879937 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.880029 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:46.882352 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:46.882354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:46.882355 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:46.882364 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:46.883800 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:46.884304 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:46.884306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:46.884306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:46.884387 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:46.884709 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:46.884710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:46.884711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:46.884850 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:46.884938 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:46.885241 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 20:56:46.885331 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465}
I20260926 20:56:46.885643 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:46.875590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46859 (local address 127.0.98.3:52693)
0926 20:56:46.876154 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:46.876238 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:46.876977 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:46.877204 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:46.877208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:46.877834 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:46.878043 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:46.878049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.878747 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.878748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:46.879615 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:46.879618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.880038 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.880039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:46.880181 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:46.880754 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:46.880763 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:46.882238 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:46.883929 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:46.883936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:46.883952 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:46.884201 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:46.884508 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:46.884509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:46.884510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:46.884631 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:46.884925 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:46.884931 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:46.885093 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 20:56:46.885215 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 20:56:46.885388 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":92,"threads_started":1}
I20260926 20:56:46.886260 11079 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46859
I20260926 20:56:46.886390 11079 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:46.886608 11079 heartbeater.cc:511] Master 127.0.98.62:46859 requested a full tablet report, sending...
I20260926 20:56:46.887066 10620 ts_manager.cc:194] Registered new tserver with Master: bdbd058609f14cf7ae5667d3543c2557 (127.0.98.3:39081)
I20260926 20:56:46.887648 10620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52693
I20260926 20:56:46.887776 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:56:46.894997 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:46.889110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59746 (local address 127.0.98.62:46859)
0926 20:56:46.889183 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:46.889185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:46.889267 (+ 82us) server_negotiation.cc:407] Connection header received
0926 20:56:46.889298 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:46.889299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:46.889309 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:46.889363 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:46.889913 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.890389 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.891079 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.891255 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.891307 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:46.892587 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:46.892589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:46.892589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:46.892596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:46.893913 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:46.894229 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:46.894230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:46.894230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:46.894282 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:46.894445 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:46.894445 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:46.894445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:46.894610 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:46.894685 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:46.894802 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 20:56:46.894863 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:56:46.896812 10620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:56:46.898592 10620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:56:46.905368 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:46.901047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41889 (local address 127.0.0.1:59956)
0926 20:56:46.901400 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:46.901446 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:46.901518 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:46.901738 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:46.901741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:46.901744 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:46.901847 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:46.901849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.903192 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.903192 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:46.904267 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:46.904269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.905058 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.905059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:46.905139 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:46.905153 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 20:56:46.905221 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:56:46.905274 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":7,"thread_start_us":56,"threads_started":1}
I20260926 20:56:46.905498 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:46.901202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41990 (local address 127.0.98.3:39081)
0926 20:56:46.901385 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:46.901390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:46.901928 (+ 538us) server_negotiation.cc:407] Connection header received
0926 20:56:46.901972 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:46.901975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:46.901991 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:46.902089 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:46.902189 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.902875 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.904459 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.905155 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.905219 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:46.905271 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:46.905351 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 20:56:46.905406 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":100,"threads_started":1}
I20260926 20:56:46.905578 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:46.901228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44642 (local address 127.0.98.2:43521)
0926 20:56:46.901516 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:46.901519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:46.901827 (+ 308us) server_negotiation.cc:407] Connection header received
0926 20:56:46.901892 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:46.901894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:46.901910 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:46.901982 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:46.902177 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.903212 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.904327 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.905110 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.905186 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:46.905244 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:46.905444 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 20:56:46.905500 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":81,"threads_started":1}
I20260926 20:56:46.905545 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:46.901047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43521 (local address 127.0.0.1:44642)
0926 20:56:46.901771 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:46.901780 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:46.901825 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:46.902060 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:46.902061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:46.902062 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:46.902101 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:46.902102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.903321 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.903322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:46.904241 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:46.904243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.905280 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.905281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:46.905361 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:46.905366 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:56:46.905414 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 20:56:46.905453 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":7,"thread_start_us":98,"threads_started":1}
I20260926 20:56:46.905831 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:46.901051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39081 (local address 127.0.0.1:41990)
0926 20:56:46.901618 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:46.901627 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:46.901698 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:46.902081 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:46.902082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:46.902084 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:46.902132 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:46.902133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.902976 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.902977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:46.904364 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:46.904366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.905217 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.905218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:46.905275 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:46.905278 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:56:46.905325 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 20:56:46.905359 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260926 20:56:46.906095 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:46.901306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59956 (local address 127.0.98.1:41889)
0926 20:56:46.901480 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:46.901485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:46.901496 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:56:46.901617 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:46.901620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:46.901637 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:46.901717 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:46.901925 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.903089 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.904361 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:46.904944 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:46.905055 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:46.905130 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:46.905952 (+ 822us) server_negotiation.cc:299] Negotiation successful
0926 20:56:46.906007 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260926 20:56:46.906805 10879 tablet_service.cc:1467] Processing CreateTablet for tablet 0fcf63b12bff433db10ba3af0a485868 (DEFAULT_TABLE table=test-table [id=3620c4efd14c4b8aa4f5041772aacfb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:46.906844 10745 tablet_service.cc:1467] Processing CreateTablet for tablet 0fcf63b12bff433db10ba3af0a485868 (DEFAULT_TABLE table=test-table [id=3620c4efd14c4b8aa4f5041772aacfb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:46.906919 11013 tablet_service.cc:1467] Processing CreateTablet for tablet 0fcf63b12bff433db10ba3af0a485868 (DEFAULT_TABLE table=test-table [id=3620c4efd14c4b8aa4f5041772aacfb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:46.907209 11013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcf63b12bff433db10ba3af0a485868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:46.907209 10745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcf63b12bff433db10ba3af0a485868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:46.907236 10879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcf63b12bff433db10ba3af0a485868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:46.909912 11096 tablet_bootstrap.cc:492] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7: Bootstrap starting.
I20260926 20:56:46.910450 11098 tablet_bootstrap.cc:492] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad: Bootstrap starting.
I20260926 20:56:46.910682 11096 tablet_bootstrap.cc:654] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:46.911192 11098 tablet_bootstrap.cc:654] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:46.911255 11096 log.cc:824] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:46.911729 11098 log.cc:824] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:46.912096 11096 tablet_bootstrap.cc:492] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7: No bootstrap required, opened a new log
I20260926 20:56:46.912118 11097 tablet_bootstrap.cc:492] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557: Bootstrap starting.
I20260926 20:56:46.912168 11096 ts_tablet_manager.cc:1397] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:56:46.912554 11098 tablet_bootstrap.cc:492] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad: No bootstrap required, opened a new log
I20260926 20:56:46.912619 11098 ts_tablet_manager.cc:1397] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:56:46.912775 11097 tablet_bootstrap.cc:654] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:46.913410 11097 log.cc:824] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:46.914251 11096 raft_consensus.cc:348] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4db3ff874644b19dc83688ee622da7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43521 } } peers { permanent_uuid: "bdbd058609f14cf7ae5667d3543c2557" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39081 } } peers { permanent_uuid: "ab5d8a2b1a464fe4b6a977c9822360ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41889 } }
I20260926 20:56:46.914455 11096 raft_consensus.cc:374] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:46.914510 11096 raft_consensus.cc:729] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4db3ff874644b19dc83688ee622da7, State: Initialized, Role: FOLLOWER
I20260926 20:56:46.914609 11097 tablet_bootstrap.cc:492] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557: No bootstrap required, opened a new log
I20260926 20:56:46.914666 11097 ts_tablet_manager.cc:1397] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:56:46.914650 11096 consensus_queue.cc:260] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4db3ff874644b19dc83688ee622da7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43521 } } peers { permanent_uuid: "bdbd058609f14cf7ae5667d3543c2557" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39081 } } peers { permanent_uuid: "ab5d8a2b1a464fe4b6a977c9822360ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41889 } }
I20260926 20:56:46.914609 11098 raft_consensus.cc:348] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4db3ff874644b19dc83688ee622da7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43521 } } peers { permanent_uuid: "bdbd058609f14cf7ae5667d3543c2557" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39081 } } peers { permanent_uuid: "ab5d8a2b1a464fe4b6a977c9822360ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41889 } }
I20260926 20:56:46.914795 11098 raft_consensus.cc:374] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:46.914842 11098 raft_consensus.cc:729] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5d8a2b1a464fe4b6a977c9822360ad, State: Initialized, Role: FOLLOWER
I20260926 20:56:46.914922 11096 ts_tablet_manager.cc:1428] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:56:46.914960 11098 consensus_queue.cc:260] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4db3ff874644b19dc83688ee622da7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43521 } } peers { permanent_uuid: "bdbd058609f14cf7ae5667d3543c2557" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39081 } } peers { permanent_uuid: "ab5d8a2b1a464fe4b6a977c9822360ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41889 } }
I20260926 20:56:46.915055 10945 heartbeater.cc:503] Master 127.0.98.62:46859 was elected leader, sending a full tablet report...
I20260926 20:56:46.915235 11098 ts_tablet_manager.cc:1428] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:56:46.915352 10811 heartbeater.cc:503] Master 127.0.98.62:46859 was elected leader, sending a full tablet report...
I20260926 20:56:46.916890 11097 raft_consensus.cc:348] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4db3ff874644b19dc83688ee622da7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43521 } } peers { permanent_uuid: "bdbd058609f14cf7ae5667d3543c2557" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39081 } } peers { permanent_uuid: "ab5d8a2b1a464fe4b6a977c9822360ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41889 } }
I20260926 20:56:46.917022 11097 raft_consensus.cc:374] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:46.917044 11097 raft_consensus.cc:729] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdbd058609f14cf7ae5667d3543c2557, State: Initialized, Role: FOLLOWER
I20260926 20:56:46.917176 11097 consensus_queue.cc:260] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4db3ff874644b19dc83688ee622da7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43521 } } peers { permanent_uuid: "bdbd058609f14cf7ae5667d3543c2557" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39081 } } peers { permanent_uuid: "ab5d8a2b1a464fe4b6a977c9822360ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41889 } }
I20260926 20:56:46.917436 11097 ts_tablet_manager.cc:1428] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:56:46.917470 11079 heartbeater.cc:503] Master 127.0.98.62:46859 was elected leader, sending a full tablet report...
W20260926 20:56:46.933171 10946 tablet.cc:2369] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:56:47.028735 10812 tablet.cc:2369] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:56:47.125372 11080 tablet.cc:2369] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:56:47.185877 11103 raft_consensus.cc:482] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:56:47.186010 11103 raft_consensus.cc:504] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4db3ff874644b19dc83688ee622da7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43521 } } peers { permanent_uuid: "bdbd058609f14cf7ae5667d3543c2557" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39081 } } peers { permanent_uuid: "ab5d8a2b1a464fe4b6a977c9822360ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41889 } }
I20260926 20:56:47.186350 11103 leader_election.cc:288] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae4db3ff874644b19dc83688ee622da7 (127.0.98.2:43521), bdbd058609f14cf7ae5667d3543c2557 (127.0.98.3:39081)
I20260926 20:56:47.190621 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:47.186645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37001 (local address 127.0.98.2:43521)
0926 20:56:47.186724 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:47.186725 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:47.186771 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:56:47.186833 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:47.186834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:47.186847 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:47.186911 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:47.187130 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:47.187892 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:47.189661 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:47.190240 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:47.190305 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:47.190316 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:47.190507 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 20:56:47.190545 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:56:47.190621 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:47.186599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55867 (local address 127.0.98.3:39081)
0926 20:56:47.186684 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:47.186687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:47.186772 (+ 85us) server_negotiation.cc:407] Connection header received
0926 20:56:47.186833 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:47.186834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:47.186844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:47.186911 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:47.187130 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:47.187892 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:47.189661 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:47.190224 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:47.190305 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:47.190316 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:47.190507 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 20:56:47.190547 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:56:47.190640 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:47.186531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39081 (local address 127.0.98.1:55867)
0926 20:56:47.186689 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:47.186702 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:47.186775 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:47.186944 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:47.186944 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:47.186945 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:47.187060 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:47.187062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:47.188014 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:47.188015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:47.189562 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:47.189565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:47.190323 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:47.190324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:47.190418 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:47.190424 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:56:47.190484 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 20:56:47.190540 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":134,"thread_start_us":76,"threads_started":1}
I20260926 20:56:47.190661 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:47.186533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43521 (local address 127.0.98.1:37001)
0926 20:56:47.186713 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:47.186723 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:47.186775 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:47.186933 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:47.186934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:47.186937 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:47.187038 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:47.187041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:47.188015 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:47.188015 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:47.189562 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:47.189565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:47.190326 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:47.190327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:47.190418 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:47.190426 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 20:56:47.190484 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:56:47.190539 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":103,"thread_start_us":70,"threads_started":1}
I20260926 20:56:47.191092 11033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcf63b12bff433db10ba3af0a485868" candidate_uuid: "ab5d8a2b1a464fe4b6a977c9822360ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdbd058609f14cf7ae5667d3543c2557" is_pre_election: true
I20260926 20:56:47.191099 10899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcf63b12bff433db10ba3af0a485868" candidate_uuid: "ab5d8a2b1a464fe4b6a977c9822360ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4db3ff874644b19dc83688ee622da7" is_pre_election: true
I20260926 20:56:47.191265 11033 raft_consensus.cc:2454] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5d8a2b1a464fe4b6a977c9822360ad in term 0.
I20260926 20:56:47.191265 10899 raft_consensus.cc:2454] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5d8a2b1a464fe4b6a977c9822360ad in term 0.
I20260926 20:56:47.191536 10701 leader_election.cc:302] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5d8a2b1a464fe4b6a977c9822360ad, bdbd058609f14cf7ae5667d3543c2557; no voters:
I20260926 20:56:47.191692 11103 raft_consensus.cc:2781] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:56:47.191756 11103 raft_consensus.cc:482] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:56:47.191784 11103 raft_consensus.cc:3037] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:47.192500 11103 raft_consensus.cc:504] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4db3ff874644b19dc83688ee622da7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43521 } } peers { permanent_uuid: "bdbd058609f14cf7ae5667d3543c2557" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39081 } } peers { permanent_uuid: "ab5d8a2b1a464fe4b6a977c9822360ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41889 } }
I20260926 20:56:47.192651 11103 leader_election.cc:288] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [CANDIDATE]: Term 1 election: Requested vote from peers ae4db3ff874644b19dc83688ee622da7 (127.0.98.2:43521), bdbd058609f14cf7ae5667d3543c2557 (127.0.98.3:39081)
I20260926 20:56:47.192813 11033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcf63b12bff433db10ba3af0a485868" candidate_uuid: "ab5d8a2b1a464fe4b6a977c9822360ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdbd058609f14cf7ae5667d3543c2557"
I20260926 20:56:47.192834 10899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcf63b12bff433db10ba3af0a485868" candidate_uuid: "ab5d8a2b1a464fe4b6a977c9822360ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4db3ff874644b19dc83688ee622da7"
I20260926 20:56:47.192904 11033 raft_consensus.cc:3037] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:47.192912 10899 raft_consensus.cc:3037] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:47.193655 10899 raft_consensus.cc:2454] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5d8a2b1a464fe4b6a977c9822360ad in term 1.
I20260926 20:56:47.193764 11033 raft_consensus.cc:2454] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5d8a2b1a464fe4b6a977c9822360ad in term 1.
I20260926 20:56:47.193835 10698 leader_election.cc:302] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5d8a2b1a464fe4b6a977c9822360ad, ae4db3ff874644b19dc83688ee622da7; no voters:
I20260926 20:56:47.193931 11103 raft_consensus.cc:2781] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:47.194116 11103 raft_consensus.cc:686] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [term 1 LEADER]: Becoming Leader. State: Replica: ab5d8a2b1a464fe4b6a977c9822360ad, State: Running, Role: LEADER
I20260926 20:56:47.194249 11103 consensus_queue.cc:237] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4db3ff874644b19dc83688ee622da7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43521 } } peers { permanent_uuid: "bdbd058609f14cf7ae5667d3543c2557" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39081 } } peers { permanent_uuid: "ab5d8a2b1a464fe4b6a977c9822360ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41889 } }
I20260926 20:56:47.195097 10620 catalog_manager.cc:5495] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5d8a2b1a464fe4b6a977c9822360ad (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ab5d8a2b1a464fe4b6a977c9822360ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4db3ff874644b19dc83688ee622da7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdbd058609f14cf7ae5667d3543c2557" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5d8a2b1a464fe4b6a977c9822360ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41889 } health_report { overall_health: HEALTHY } } }
I20260926 20:56:47.259872 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:47.257722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59970 (local address 127.0.98.1:41889)
0926 20:56:47.257791 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:47.257793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:47.257800 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:56:47.257834 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:47.257835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:47.257854 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:47.257921 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:47.258050 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:47.258508 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:47.259275 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:47.259466 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:47.259558 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:47.259627 (+ 69us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:47.259723 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:47.259779 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:47.259795 (+ 16us) server_negotiation.cc:299] Negotiation successful
0926 20:56:47.259812 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:56:47.261801 10899 raft_consensus.cc:1261] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7 [term 1 FOLLOWER]: Refusing update from remote peer ab5d8a2b1a464fe4b6a977c9822360ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:47.261801 11033 raft_consensus.cc:1261] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557 [term 1 FOLLOWER]: Refusing update from remote peer ab5d8a2b1a464fe4b6a977c9822360ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:47.262215 11107 consensus_queue.cc:1035] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdbd058609f14cf7ae5667d3543c2557" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:47.262368 11103 consensus_queue.cc:1035] T 0fcf63b12bff433db10ba3af0a485868 P ab5d8a2b1a464fe4b6a977c9822360ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4db3ff874644b19dc83688ee622da7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:47.266052 11112 mvcc.cc:205] Tried to move back new op lower bound from 7333708624941621248 to 7333708624668307456. Current Snapshot: MvccSnapshot[applied={T|T < 7333708624941621248 or (T in {7333708624941621248})}]
I20260926 20:56:47.267407 10620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:47.267515 10620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:47.268257 10620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd023341ead3459e83a74a7b20b75e63: Sending DeleteTablet for 3 replicas of tablet 0fcf63b12bff433db10ba3af0a485868
I20260926 20:56:47.268674 10879 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fcf63b12bff433db10ba3af0a485868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44642
I20260926 20:56:47.268877 11013 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fcf63b12bff433db10ba3af0a485868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41990
I20260926 20:56:47.268911 11120 tablet_replica.cc:330] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7: stopping tablet replica
I20260926 20:56:47.268891 10745 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fcf63b12bff433db10ba3af0a485868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59956
I20260926 20:56:47.269014 11120 raft_consensus.cc:2229] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:47.269130 11120 raft_consensus.cc:2258] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:47.269322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 10682
I20260926 20:56:47.270071 11121 tablet_replica.cc:330] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557: stopping tablet replica
I20260926 20:56:47.270179 11121 raft_consensus.cc:2229] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:47.270279 11121 raft_consensus.cc:2258] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:47.270798 11121 ts_tablet_manager.cc:1905] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:47.272159 11120 ts_tablet_manager.cc:1905] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:47.272372 11121 ts_tablet_manager.cc:1918] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:47.272512 11121 log.cc:1196] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/0fcf63b12bff433db10ba3af0a485868
I20260926 20:56:47.272801 11121 ts_tablet_manager.cc:1939] T 0fcf63b12bff433db10ba3af0a485868 P bdbd058609f14cf7ae5667d3543c2557: Deleting consensus metadata
I20260926 20:56:47.273277 10607 catalog_manager.cc:4841] TS bdbd058609f14cf7ae5667d3543c2557 (127.0.98.3:39081): tablet 0fcf63b12bff433db10ba3af0a485868 (table test-table [id=3620c4efd14c4b8aa4f5041772aacfb1]) successfully deleted
I20260926 20:56:47.273856 11120 ts_tablet_manager.cc:1918] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:47.273916 11120 log.cc:1196] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/0fcf63b12bff433db10ba3af0a485868
I20260926 20:56:47.274149 11120 ts_tablet_manager.cc:1939] T 0fcf63b12bff433db10ba3af0a485868 P ae4db3ff874644b19dc83688ee622da7: Deleting consensus metadata
I20260926 20:56:47.274556 10604 catalog_manager.cc:4841] TS ae4db3ff874644b19dc83688ee622da7 (127.0.98.2:43521): tablet 0fcf63b12bff433db10ba3af0a485868 (table test-table [id=3620c4efd14c4b8aa4f5041772aacfb1]) successfully deleted
W20260926 20:56:47.274909 10606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41889 (error 108)
I20260926 20:56:47.275417 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:47.275276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41889 (local address 127.0.0.1:59978)
0926 20:56:47.275342 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:47.275390 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 20:56:47.275532 10606 catalog_manager.cc:4584] TS ab5d8a2b1a464fe4b6a977c9822360ad (127.0.98.1:41889): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0fcf63b12bff433db10ba3af0a485868: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41889: connect: Connection refused (error 111)
I20260926 20:56:47.275563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 10816
I20260926 20:56:47.280190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 10950
I20260926 20:56:47.284870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 10588
2026-09-26T20:56:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2257 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:56:47 dist-test-slave-fl7g krb5kdc[11126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:56:47 dist-test-slave-fl7g krb5kdc[11126](info): set up 2 sockets
Sep 26 20:56:47 dist-test-slave-fl7g krb5kdc[11126](info): commencing operation
krb5kdc: starting...
W20260926 20:56:48.361687 392 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:56:48 dist-test-slave-fl7g krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:56:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 20:56:48.537504 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskVxazDW/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/postgres -l logfile start
2026-09-26 20:56:51.329 UTC [11156] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 20:56:51.329 UTC [11156] LOG: listening on IPv4 address "127.0.98.20", port 58641
2026-09-26 20:56:51.334 UTC [11156] LOG: listening on Unix socket "/tmp/.s.PGSQL.58641"
2026-09-26 20:56:51.370 UTC [11159] LOG: database system was shut down at 2026-09-26 20:56:49 UTC
2026-09-26 20:56:51.375 UTC [11156] LOG: database system is ready to accept connections
I20260926 20:56:52.329651 392 mini_postgres.cc:96] Postgres bound to 58641
2026-09-26 20:56:52.335 UTC [11167] FATAL: database "slave" does not exist
127.0.98.20:58641 - accepting connections
I20260926 20:56:52.336584 392 mini_ranger.cc:161] Starting Ranger...
I20260926 20:56:52.352880 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 20:56:52.939455 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 20:56:52.939556 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ranger-admin
2026-09-26 20:56:53,236 [I] DB FLAVOR :POSTGRES
2026-09-26 20:56:53,236 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:56:53,236 [I] Checking connection..
2026-09-26 20:56:53,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 20:56:53,515 [I] Checking connection passed.
2026-09-26 20:56:53,516 [I] --------- Verifying version history table ---------
2026-09-26 20:56:53,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:56:53,763 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:56:53,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:56:54,028 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:56:54,028 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 20:56:54,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 20:56:54.262 UTC [11231] WARNING: there is no transaction in progress
2026-09-26 20:56:54,268 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 20:56:54,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:56:54,524 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 20:56:54,524 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:56:54,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 20:56:54,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 20:56:55,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fl7g.c.gcp-upstream.internal','N') ;"
2026-09-26 20:56:55,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:56:55,550 [I] Table x_portal_user does not exist in database ranger
2026-09-26 20:56:55,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:56:55,807 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 20:56:55,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:56:56,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:56:56,304 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 20:56:56,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 20:56:57.352 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 20:56:57.371 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 20:56:57.390 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 20:56:57.408 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 20:56:57.426 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 20:56:57.445 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 20:56:57.505 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 20:56:57.513 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 20:56:57.530 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 20:56:57.541 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 20:56:57.553 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 20:56:57.568 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 20:56:57.579 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 20:56:57.587 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 20:56:57.595 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 20:56:57.603 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 20:56:58,369 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 20:56:58,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 20:56:58,607 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 20:56:58,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:56:58,872 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 20:56:58,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:56:59,127 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 20:56:59,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:56:59,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:56:59,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fl7g.c.gcp-upstream.internal';"
2026-09-26 20:56:59,896 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 20:56:59,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 20:57:00,138 [I] DB_PATCHES have already been applied
I20260926 20:57:00.144315 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskVxazDW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 20:57:00.144390 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 20:57:00.147681 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:36995
Sep 26, 2026 8:57:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:57:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:57:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 8:57:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 8:57:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 8:57:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 8:57:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 20:57:01.258785 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.922s user 0.004s sys 0.005s
Sep 26, 2026 8:57:21 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 8:57:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 8:57:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 8:57:21 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 8:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 8:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 8:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 8:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 8:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 8:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 8:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 8:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 8:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 8:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 8:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 8:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 8:57:21 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 8:57:22 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 20:57:22.880273 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 20:57:22 dist-test-slave-fl7g krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456242, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 20:57:22.970824 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 20:57:22.980638 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 20:57:24.003868 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 20:57:24.003960 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ranger-kms
I20260926 20:57:24.003980 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:58641
2026-09-26 20:57:24,185 [I] DB FLAVOR :POSTGRES
2026-09-26 20:57:24,186 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:57:24,186 [I] Checking connection
2026-09-26 20:57:24,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 20:57:24,462 [I] connection success
2026-09-26 20:57:24,463 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 20:57:24,463 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 20:57:24,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 20:57:24,718 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 20:57:24,718 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:57:24,718 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 20:57:24,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58641/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 20:57:25,000 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 20:57:25.481402 392 mini_ranger_kms.cc:310] Created kms service
I20260926 20:57:25.659032 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 20:57:25.707836 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 20:57:25 dist-test-slave-fl7g krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456242, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:57:26.107529 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 20:57:26.107658 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 20:57:26.107693 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 20:57:26.111754 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11519
I20260926 20:57:26.111887 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:42891
Sep 26, 2026 8:57:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:57:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:57:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 8:57:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 8:57:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 8:57:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 20:57:28.351249 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.380s user 0.000s sys 0.163s
I20260926 20:57:28.351363 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 20:57:28.351436 392 mini_ranger_kms.cc:401] 127.0.98.20:42891/kms/v1/keys
Sep 26 20:57:28 dist-test-slave-fl7g krb5kdc[11126](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790456248, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 20:57:29 dist-test-slave-fl7g krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456248, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 20:57:29 dist-test-slave-fl7g krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456248, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 20:57:31 dist-test-slave-fl7g krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:57:31.808569 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41753
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41753
--ranger_config_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42891/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:31.921751 11570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:31.921981 11570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:31.922003 11570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:31.923264 11570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:31.923317 11570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:31.923332 11570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:31.923345 11570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:31.923357 11570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:31.925065 11570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:42891/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41753
--ranger_config_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:57:31.925335 11570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:31.925576 11570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:31.931968 11578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:31.932054 11570 server_base.cc:1034] running on GCE node
W20260926 20:57:31.932211 11575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:31.932107 11576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:31.932678 11570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:31.932996 11570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:31.934167 11570 hybrid_clock.cc:648] HybridClock initialized: now 1790456251934155 us; error 34 us; skew 500 ppm
Sep 26 20:57:31 dist-test-slave-fl7g krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:31.936733 11570 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:57:31.937196 11570 webserver.cc:460] Webserver started at http://127.0.98.62:45577/ using document root <none> and password file <none>
I20260926 20:57:31.937430 11570 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:31.937491 11570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:31.937655 11570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:57:31 dist-test-slave-fl7g krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:57:31.985623 11570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "134b9796cf5f418fb72150f209d35a96"
format_stamp: "Formatted at 2026-09-26 20:57:31 on dist-test-slave-fl7g"
server_key: "242e7ebbf487b9465ad94d586f54ad6a"
server_key_iv: "4dd0eb25e5d2b2b532b4546334512eeb"
server_key_version: "kuduclusterkey@0"
I20260926 20:57:31.986083 11570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "134b9796cf5f418fb72150f209d35a96"
format_stamp: "Formatted at 2026-09-26 20:57:31 on dist-test-slave-fl7g"
server_key: "242e7ebbf487b9465ad94d586f54ad6a"
server_key_iv: "4dd0eb25e5d2b2b532b4546334512eeb"
server_key_version: "kuduclusterkey@0"
I20260926 20:57:31.987617 11570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:57:32.000838 11585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:32.001093 11570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:57:32.001200 11570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "134b9796cf5f418fb72150f209d35a96"
format_stamp: "Formatted at 2026-09-26 20:57:31 on dist-test-slave-fl7g"
server_key: "242e7ebbf487b9465ad94d586f54ad6a"
server_key_iv: "4dd0eb25e5d2b2b532b4546334512eeb"
server_key_version: "kuduclusterkey@0"
I20260926 20:57:32.001333 11570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:32.012521 11570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:32.014544 11570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:32.014668 11570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:32.020294 11570 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41753
I20260926 20:57:32.020308 11637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41753 every 8 connection(s)
I20260926 20:57:32.020777 11570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:32.021663 11638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:32.025000 11638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96: Bootstrap starting.
I20260926 20:57:32.025730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 11570
I20260926 20:57:32.025768 11638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:32.025862 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:57:32.026080 392 external_mini_cluster.cc:1428] Setting key 0e045491dead936c70f36772457e8740
I20260926 20:57:32.026062 11638 log.cc:824] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:32.027180 11638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96: No bootstrap required, opened a new log
I20260926 20:57:32.029230 11638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134b9796cf5f418fb72150f209d35a96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41753 } }
I20260926 20:57:32.029424 11638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:32.029456 11638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 134b9796cf5f418fb72150f209d35a96, State: Initialized, Role: FOLLOWER
I20260926 20:57:32.029579 11638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134b9796cf5f418fb72150f209d35a96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41753 } }
I20260926 20:57:32.029688 11638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:32.029736 11638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:32.029793 11638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:32.030541 11638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134b9796cf5f418fb72150f209d35a96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41753 } }
I20260926 20:57:32.030694 11638 leader_election.cc:302] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 134b9796cf5f418fb72150f209d35a96; no voters:
I20260926 20:57:32.030898 11638 leader_election.cc:288] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:32.030990 11643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:32.031174 11638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:32.031198 11643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96 [term 1 LEADER]: Becoming Leader. State: Replica: 134b9796cf5f418fb72150f209d35a96, State: Running, Role: LEADER
I20260926 20:57:32.031430 11643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134b9796cf5f418fb72150f209d35a96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41753 } }
I20260926 20:57:32.031841 11643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "134b9796cf5f418fb72150f209d35a96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134b9796cf5f418fb72150f209d35a96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41753 } } }
I20260926 20:57:32.032016 11643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:32.032212 11643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 134b9796cf5f418fb72150f209d35a96. Latest consensus state: current_term: 1 leader_uuid: "134b9796cf5f418fb72150f209d35a96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134b9796cf5f418fb72150f209d35a96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41753 } } }
I20260926 20:57:32.032328 11643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:32.032552 11654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:32.032604 11638 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:57:32.033181 11654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 20:57:32 dist-test-slave-fl7g krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:57:32.042711 11654 catalog_manager.cc:1347] Generated new cluster ID: c093b40f6d4d4304884203c37eb9489e
I20260926 20:57:32.042958 11654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:32.043896 11642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:32.026716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.0.98.62:41753)
0926 20:57:32.026898 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:32.026910 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:32.026948 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:57:32.027496 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:32.027503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:32.027560 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:32.027717 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:32.028484 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:32.030045 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:32.031173 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:32.031846 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:32.031956 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:32.035212 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:32.035218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:32.035218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:32.035233 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:32.040531 (+ 5298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:32.041142 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:32.041146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:32.041146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:32.041344 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:32.041601 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:32.041602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:32.041602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:32.042196 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:32.042422 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:32.043006 (+ 584us) server_negotiation.cc:299] Negotiation successful
0926 20:57:32.043387 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260926 20:57:32.084790 11654 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:32.085630 11654 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:32.097517 11654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96: Generated new TSK 0
I20260926 20:57:32.097797 11654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:57:32.853336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42891/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41753
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:32.968956 11683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:32.969223 11683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:32.969266 11683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:32.970543 11683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:32.970623 11683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:32.970666 11683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:57:32.972260 11683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:42891/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:57:32.972582 11683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:32.972831 11683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:32.979398 11691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:32.979408 11688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:32.979508 11689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:32.979635 11683 server_base.cc:1034] running on GCE node
I20260926 20:57:32.979954 11683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:32.980234 11683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:32.981416 11683 hybrid_clock.cc:648] HybridClock initialized: now 1790456252981389 us; error 46 us; skew 500 ppm
Sep 26 20:57:32 dist-test-slave-fl7g krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:32.984640 11683 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:57:32.985236 11683 webserver.cc:460] Webserver started at http://127.0.98.1:40145/ using document root <none> and password file <none>
I20260926 20:57:32.985488 11683 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:32.985558 11683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:32.985710 11683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:57:32 dist-test-slave-fl7g krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:57:33.000892 11683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "ddb27858eb984d79a4a766a8cec180d5"
format_stamp: "Formatted at 2026-09-26 20:57:32 on dist-test-slave-fl7g"
server_key: "ce347858b6b2f7414d1da04a07a37e26"
server_key_iv: "b65cd1df934fb130639aff30a661ce17"
server_key_version: "kuduclusterkey@0"
I20260926 20:57:33.001466 11683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ddb27858eb984d79a4a766a8cec180d5"
format_stamp: "Formatted at 2026-09-26 20:57:32 on dist-test-slave-fl7g"
server_key: "ce347858b6b2f7414d1da04a07a37e26"
server_key_iv: "b65cd1df934fb130639aff30a661ce17"
server_key_version: "kuduclusterkey@0"
I20260926 20:57:33.004891 11683 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:57:33.017236 11699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:33.017493 11683 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:57:33.017593 11683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "ddb27858eb984d79a4a766a8cec180d5"
format_stamp: "Formatted at 2026-09-26 20:57:32 on dist-test-slave-fl7g"
server_key: "ce347858b6b2f7414d1da04a07a37e26"
server_key_iv: "b65cd1df934fb130639aff30a661ce17"
server_key_version: "kuduclusterkey@0"
I20260926 20:57:33.017699 11683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:33.035001 11683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:33.037729 11683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:33.037896 11683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:33.038115 11683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:57:33.038472 11683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:57:33.038514 11683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:33.038545 11683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:57:33.038604 11683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:33.046540 11683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33189
I20260926 20:57:33.047077 11683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:57:33.049840 11813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33189 every 8 connection(s)
I20260926 20:57:33.051118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 11683
I20260926 20:57:33.051236 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:57:33.051460 392 external_mini_cluster.cc:1428] Setting key e41e52729c98dd6b67378a602d89540c
Sep 26 20:57:33 dist-test-slave-fl7g krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:57:33.062240 11816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:33.049232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41753 (local address 127.0.98.1:52955)
0926 20:57:33.050270 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:33.050313 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:33.050970 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:33.051644 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:33.051649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:33.051901 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:33.052201 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:33.052207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.053256 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.053257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:33.054558 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:33.054561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.055108 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.055109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:33.055268 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.056403 (+ 1135us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:33.056461 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:33.058202 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:33.060346 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:33.060354 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:33.060356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:33.060722 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:33.060964 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:33.060965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:33.060966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:33.061110 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:33.061454 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:33.061460 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:33.061658 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 20:57:33.061781 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 20:57:33.061962 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":993,"thread_start_us":91,"threads_started":1}
I20260926 20:57:33.062908 11817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:33.049536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52955 (local address 127.0.98.62:41753)
0926 20:57:33.051260 (+ 1724us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:33.051319 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:33.051331 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:57:33.051363 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:33.051364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:33.051380 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:33.051451 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:33.052320 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.053142 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.054664 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.054977 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.055040 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.058339 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:33.058342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:33.058342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:33.058351 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:33.060230 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:33.060817 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:33.060819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:33.060819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:33.060888 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:33.061204 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:33.061205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:33.061205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:33.061361 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:33.061426 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:33.062711 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0926 20:57:33.062807 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1620,"thread_start_us":79,"threads_started":1}
I20260926 20:57:33.063742 11814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41753
I20260926 20:57:33.063844 11814 heartbeater.cc:464] Registering TS with master...
I20260926 20:57:33.064045 11814 heartbeater.cc:511] Master 127.0.98.62:41753 requested a full tablet report, sending...
I20260926 20:57:33.064719 11602 ts_manager.cc:194] Registered new tserver with Master: ddb27858eb984d79a4a766a8cec180d5 (127.0.98.1:33189)
I20260926 20:57:33.065619 11602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52955
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:57:33.095943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42891/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41753
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:33.230059 11820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:33.230458 11820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:33.230551 11820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:33.232582 11820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:33.232713 11820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:33.232810 11820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:57:33.235316 11820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:42891/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:57:33.235862 11820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:33.236200 11820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:33.243031 11828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:33.243453 11827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:33.243688 11830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:33.245028 11820 server_base.cc:1034] running on GCE node
I20260926 20:57:33.245250 11820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:33.245510 11820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:33.246691 11820 hybrid_clock.cc:648] HybridClock initialized: now 1790456253246670 us; error 35 us; skew 500 ppm
Sep 26 20:57:33 dist-test-slave-fl7g krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:33.250119 11820 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:57:33.250662 11820 webserver.cc:460] Webserver started at http://127.0.98.2:43923/ using document root <none> and password file <none>
I20260926 20:57:33.250905 11820 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:33.250954 11820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:33.251076 11820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:57:33 dist-test-slave-fl7g krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:57:33.266309 11820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "0532a89e1723473b9a18462259472807"
format_stamp: "Formatted at 2026-09-26 20:57:33 on dist-test-slave-fl7g"
server_key: "75ef72f0682dc2018999ea239ec50d85"
server_key_iv: "f839f8fa1d72d875d97b14d8e9cc858b"
server_key_version: "kuduclusterkey@0"
I20260926 20:57:33.266925 11820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0532a89e1723473b9a18462259472807"
format_stamp: "Formatted at 2026-09-26 20:57:33 on dist-test-slave-fl7g"
server_key: "75ef72f0682dc2018999ea239ec50d85"
server_key_iv: "f839f8fa1d72d875d97b14d8e9cc858b"
server_key_version: "kuduclusterkey@0"
I20260926 20:57:33.268878 11820 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:57:33.283876 11837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:33.284163 11820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:57:33.284255 11820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "0532a89e1723473b9a18462259472807"
format_stamp: "Formatted at 2026-09-26 20:57:33 on dist-test-slave-fl7g"
server_key: "75ef72f0682dc2018999ea239ec50d85"
server_key_iv: "f839f8fa1d72d875d97b14d8e9cc858b"
server_key_version: "kuduclusterkey@0"
I20260926 20:57:33.284348 11820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 26 20:57:33 dist-test-slave-fl7g krb5kdc[11126](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790456253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:33.299986 11820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:33.302671 11820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:33.302899 11820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:33.303185 11820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:57:33.303646 11820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:57:33.303737 11820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:33.303805 11820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:57:33.303861 11820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:33.310899 11820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37175
I20260926 20:57:33.311293 11950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37175 every 8 connection(s)
I20260926 20:57:33.311424 11820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:57:33.318748 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 11820
I20260926 20:57:33.318871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:57:33.319061 392 external_mini_cluster.cc:1428] Setting key 5fc558da4207e82ba3b3c009b4ef27af
Sep 26 20:57:33 dist-test-slave-fl7g krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:57:33.323930 11817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:33.313054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51249 (local address 127.0.98.62:41753)
0926 20:57:33.313104 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:33.313107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:33.313343 (+ 236us) server_negotiation.cc:407] Connection header received
0926 20:57:33.314041 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:33.314043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:33.314055 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:33.314087 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:33.314911 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.315671 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.317195 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.317497 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.317536 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.320500 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:33.320501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:33.320502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:33.320511 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:33.322340 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:33.322874 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:33.322875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:33.322876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:33.322932 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:33.323203 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:33.323204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:33.323204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:33.323332 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:33.323371 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:33.323743 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 20:57:33.323831 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:57:33.324728 11953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:33.313060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41753 (local address 127.0.98.2:51249)
0926 20:57:33.313221 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:33.313262 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:33.313916 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:33.314282 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:33.314286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:33.314534 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:33.314803 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:33.314808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.315789 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.315790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:33.317080 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:33.317083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.317623 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.317623 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:33.317762 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.318396 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:33.318406 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:33.320341 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:33.322436 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:33.322443 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:33.322446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:33.322790 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:33.322997 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:33.322998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:33.322998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:33.323122 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:33.323413 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:33.323419 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:33.323595 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 20:57:33.324291 (+ 696us) client_negotiation.cc:240] Negotiation successful
0926 20:57:33.324496 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260926 20:57:33.325379 11951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41753
I20260926 20:57:33.325472 11951 heartbeater.cc:464] Registering TS with master...
I20260926 20:57:33.325656 11951 heartbeater.cc:511] Master 127.0.98.62:41753 requested a full tablet report, sending...
I20260926 20:57:33.326061 11602 ts_manager.cc:194] Registered new tserver with Master: 0532a89e1723473b9a18462259472807 (127.0.98.2:37175)
I20260926 20:57:33.326542 11602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51249
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:57:33.364079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42891/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41753
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:33.476733 11956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:33.477126 11956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:33.477231 11956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:33.479212 11956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:33.479359 11956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:33.479452 11956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:57:33.481950 11956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:42891/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:57:33.482688 11956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:33.483037 11956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:33.489921 11963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:33.490391 11965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:33.491382 11956 server_base.cc:1034] running on GCE node
W20260926 20:57:33.497185 11962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:33.497640 11956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:33.497993 11956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:33.499181 11956 hybrid_clock.cc:648] HybridClock initialized: now 1790456253499161 us; error 38 us; skew 500 ppm
Sep 26 20:57:33 dist-test-slave-fl7g krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:33.502934 11956 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:57:33.503516 11956 webserver.cc:460] Webserver started at http://127.0.98.3:45477/ using document root <none> and password file <none>
I20260926 20:57:33.503912 11956 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:33.504031 11956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:33.504230 11956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:57:33 dist-test-slave-fl7g krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:57:33.525488 11956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "26218a6553084295b115593e3bf5c117"
format_stamp: "Formatted at 2026-09-26 20:57:33 on dist-test-slave-fl7g"
server_key: "4c8cbaf6d62951edfa530bb3211ab072"
server_key_iv: "6f0a748b827076210aee47ccbe8d05d2"
server_key_version: "kuduclusterkey@0"
I20260926 20:57:33.526141 11956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "26218a6553084295b115593e3bf5c117"
format_stamp: "Formatted at 2026-09-26 20:57:33 on dist-test-slave-fl7g"
server_key: "4c8cbaf6d62951edfa530bb3211ab072"
server_key_iv: "6f0a748b827076210aee47ccbe8d05d2"
server_key_version: "kuduclusterkey@0"
I20260926 20:57:33.529121 11956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:57:33.546074 11972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:33.546368 11956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:57:33.546470 11956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "26218a6553084295b115593e3bf5c117"
format_stamp: "Formatted at 2026-09-26 20:57:33 on dist-test-slave-fl7g"
server_key: "4c8cbaf6d62951edfa530bb3211ab072"
server_key_iv: "6f0a748b827076210aee47ccbe8d05d2"
server_key_version: "kuduclusterkey@0"
I20260926 20:57:33.546588 11956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:33.572497 11956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:33.575352 11956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:33.575608 11956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:33.575908 11956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:57:33.576449 11956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:57:33.576599 11956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:33.576686 11956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:57:33.576746 11956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:33.583698 11956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35269
I20260926 20:57:33.584230 11956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:57:33.586860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 11956
I20260926 20:57:33.586969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:57:33.587229 392 external_mini_cluster.cc:1428] Setting key 66a690dcfc037bc7d07921990b309a58
I20260926 20:57:33.590075 12085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35269 every 8 connection(s)
Sep 26 20:57:33 dist-test-slave-fl7g krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:57:33.599834 11817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:33.590704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58943 (local address 127.0.98.62:41753)
0926 20:57:33.590753 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:33.590756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:33.591057 (+ 301us) server_negotiation.cc:407] Connection header received
0926 20:57:33.591614 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:33.591616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:33.591629 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:33.591670 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:33.592269 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.592880 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.593919 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.594149 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.594243 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.596603 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:33.596605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:33.596605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:33.596613 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:33.598149 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:33.598666 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:33.598668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:33.598668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:33.598755 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:33.599037 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:33.599038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:33.599038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:33.599204 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:33.599272 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:33.599688 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 20:57:33.599763 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:57:33.600080 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:33.590710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41753 (local address 127.0.98.3:58943)
0926 20:57:33.590906 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:33.590953 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:33.591499 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:33.591743 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:33.591748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:33.591967 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:33.592174 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:33.592179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.592986 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.592987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:33.593823 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:33.593825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.594256 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.594258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:33.594394 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.595029 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:33.595038 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:33.596496 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:33.598273 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:33.598281 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:33.598284 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:33.598533 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:33.598824 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:33.598824 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:33.598825 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:33.598979 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:33.599335 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:33.599343 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:33.599543 (+ 200us) client_negotiation.cc:769] Sending connection context
0926 20:57:33.599671 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 20:57:33.599839 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260926 20:57:33.600678 12086 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41753
I20260926 20:57:33.600781 12086 heartbeater.cc:464] Registering TS with master...
I20260926 20:57:33.600991 12086 heartbeater.cc:511] Master 127.0.98.62:41753 requested a full tablet report, sending...
I20260926 20:57:33.601397 11602 ts_manager.cc:194] Registered new tserver with Master: 26218a6553084295b115593e3bf5c117 (127.0.98.3:35269)
I20260926 20:57:33.601797 11602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58943
I20260926 20:57:33.610450 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:57:33.620327 11817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:33.612315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.0.98.62:41753)
0926 20:57:33.612370 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:33.612372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:33.612476 (+ 104us) server_negotiation.cc:407] Connection header received
0926 20:57:33.612555 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:33.612556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:33.612567 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:33.612649 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:33.613238 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.613998 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.614732 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.614998 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.615068 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.616340 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:33.616342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:33.616342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:33.616351 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:33.618645 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:33.619023 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:33.619025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:33.619025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:33.619101 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:33.619345 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:33.619346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:33.619347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:33.619514 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:33.619641 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:33.620115 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 20:57:33.620201 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 20:57:33.622720 11602 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:57:33.625492 11602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:57:33.642940 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:33.635434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33189 (local address 127.0.0.1:49046)
0926 20:57:33.637574 (+ 2140us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:33.637628 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:33.637721 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:33.638003 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:33.638007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:33.638009 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:33.638085 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:33.638087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.639726 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.639727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:33.641473 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:33.641476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.642581 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.642582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:33.642685 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.642702 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 20:57:33.642778 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:57:33.642842 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2107,"thread_start_us":117,"threads_started":1}
I20260926 20:57:33.643396 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:33.636277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49046 (local address 127.0.98.1:33189)
0926 20:57:33.637259 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:33.637264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:33.637734 (+ 470us) server_negotiation.cc:407] Connection header received
0926 20:57:33.637809 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:33.637812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:33.637830 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:33.637916 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:33.638162 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.639618 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.641563 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.642391 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.642476 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.642543 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:33.643281 (+ 738us) server_negotiation.cc:299] Negotiation successful
0926 20:57:33.643332 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":943,"thread_start_us":58,"threads_started":1}
I20260926 20:57:33.645063 11748 tablet_service.cc:1467] Processing CreateTablet for tablet 5f54c16ecf1944deb480c2b1da864060 (DEFAULT_TABLE table=test-table [id=d8ffe5c6cdec40e4a055ea7b825669f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:57:33.645434 11748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f54c16ecf1944deb480c2b1da864060. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:33.649153 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:33.635732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35269 (local address 127.0.0.1:39284)
0926 20:57:33.637936 (+ 2204us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:33.637946 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:33.637996 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:33.638297 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:33.638297 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:33.638299 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:33.638355 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:33.638356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.644471 (+ 6115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.644472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:33.645712 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:33.645714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.648862 (+ 3148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.648863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:33.648943 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.648948 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:57:33.648999 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 20:57:33.649050 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2189,"thread_start_us":85,"threads_started":1}
I20260926 20:57:33.649556 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:33.636506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37175 (local address 127.0.0.1:44340)
0926 20:57:33.643741 (+ 7235us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:33.643752 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:33.643792 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:33.644439 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:33.644441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:33.644443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:33.644477 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:33.644479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.645673 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.645674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:33.646724 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:33.646726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.649315 (+ 2589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.649316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:33.649376 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.649379 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:57:33.649433 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:57:33.649479 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7221,"mutex_wait_us":3,"thread_start_us":64,"threads_started":1}
I20260926 20:57:33.649829 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:33.636725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.0.98.2:37175)
0926 20:57:33.637884 (+ 1159us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:33.637889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:33.643855 (+ 5966us) server_negotiation.cc:407] Connection header received
0926 20:57:33.644190 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:33.644194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:33.644212 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:33.644290 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:33.644560 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.645574 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.646848 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.647631 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.647706 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.647794 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:33.649679 (+ 1885us) server_negotiation.cc:299] Negotiation successful
0926 20:57:33.649731 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1120,"thread_start_us":88,"threads_started":1}
I20260926 20:57:33.649981 12104 tablet_bootstrap.cc:492] T 5f54c16ecf1944deb480c2b1da864060 P ddb27858eb984d79a4a766a8cec180d5: Bootstrap starting.
I20260926 20:57:33.650559 11885 tablet_service.cc:1467] Processing CreateTablet for tablet 5f54c16ecf1944deb480c2b1da864060 (DEFAULT_TABLE table=test-table [id=d8ffe5c6cdec40e4a055ea7b825669f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:57:33.650743 12104 tablet_bootstrap.cc:654] T 5f54c16ecf1944deb480c2b1da864060 P ddb27858eb984d79a4a766a8cec180d5: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:33.651350 11885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f54c16ecf1944deb480c2b1da864060. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:33.651499 12104 log.cc:824] T 5f54c16ecf1944deb480c2b1da864060 P ddb27858eb984d79a4a766a8cec180d5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:33.652442 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:33.636083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39284 (local address 127.0.98.3:35269)
0926 20:57:33.637495 (+ 1412us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:33.637501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:33.638071 (+ 570us) server_negotiation.cc:407] Connection header received
0926 20:57:33.638117 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:33.638120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:33.638136 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:33.638207 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:33.638437 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.644222 (+ 5785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.647859 (+ 3637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.648669 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.648744 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.648817 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:33.652270 (+ 3453us) server_negotiation.cc:299] Negotiation successful
0926 20:57:33.652330 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1367,"thread_start_us":78,"threads_started":1}
I20260926 20:57:33.652562 12104 tablet_bootstrap.cc:492] T 5f54c16ecf1944deb480c2b1da864060 P ddb27858eb984d79a4a766a8cec180d5: No bootstrap required, opened a new log
I20260926 20:57:33.652635 12104 ts_tablet_manager.cc:1397] T 5f54c16ecf1944deb480c2b1da864060 P ddb27858eb984d79a4a766a8cec180d5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:57:33.655290 12104 raft_consensus.cc:348] T 5f54c16ecf1944deb480c2b1da864060 P ddb27858eb984d79a4a766a8cec180d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0532a89e1723473b9a18462259472807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37175 } } peers { permanent_uuid: "ddb27858eb984d79a4a766a8cec180d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33189 } } peers { permanent_uuid: "26218a6553084295b115593e3bf5c117" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35269 } }
I20260926 20:57:33.655467 12104 raft_consensus.cc:374] T 5f54c16ecf1944deb480c2b1da864060 P ddb27858eb984d79a4a766a8cec180d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:33.655507 12104 raft_consensus.cc:729] T 5f54c16ecf1944deb480c2b1da864060 P ddb27858eb984d79a4a766a8cec180d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb27858eb984d79a4a766a8cec180d5, State: Initialized, Role: FOLLOWER
I20260926 20:57:33.655565 12106 tablet_bootstrap.cc:492] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807: Bootstrap starting.
I20260926 20:57:33.656594 12106 tablet_bootstrap.cc:654] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:33.656344 12001 tablet_service.cc:1467] Processing CreateTablet for tablet 5f54c16ecf1944deb480c2b1da864060 (DEFAULT_TABLE table=test-table [id=d8ffe5c6cdec40e4a055ea7b825669f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:57:33.656929 12106 log.cc:824] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:33.657040 12001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f54c16ecf1944deb480c2b1da864060. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:33.657670 12106 tablet_bootstrap.cc:492] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807: No bootstrap required, opened a new log
I20260926 20:57:33.657728 12106 ts_tablet_manager.cc:1397] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:57:33.659726 12106 raft_consensus.cc:348] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0532a89e1723473b9a18462259472807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37175 } } peers { permanent_uuid: "ddb27858eb984d79a4a766a8cec180d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33189 } } peers { permanent_uuid: "26218a6553084295b115593e3bf5c117" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35269 } }
I20260926 20:57:33.659905 12106 raft_consensus.cc:374] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:33.659947 12106 raft_consensus.cc:729] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0532a89e1723473b9a18462259472807, State: Initialized, Role: FOLLOWER
I20260926 20:57:33.660068 12106 consensus_queue.cc:260] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0532a89e1723473b9a18462259472807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37175 } } peers { permanent_uuid: "ddb27858eb984d79a4a766a8cec180d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33189 } } peers { permanent_uuid: "26218a6553084295b115593e3bf5c117" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35269 } }
I20260926 20:57:33.660331 12106 ts_tablet_manager.cc:1428] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:57:33.660843 11951 heartbeater.cc:503] Master 127.0.98.62:41753 was elected leader, sending a full tablet report...
I20260926 20:57:33.660842 12104 consensus_queue.cc:260] T 5f54c16ecf1944deb480c2b1da864060 P ddb27858eb984d79a4a766a8cec180d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0532a89e1723473b9a18462259472807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37175 } } peers { permanent_uuid: "ddb27858eb984d79a4a766a8cec180d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33189 } } peers { permanent_uuid: "26218a6553084295b115593e3bf5c117" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35269 } }
I20260926 20:57:33.661154 12104 ts_tablet_manager.cc:1428] T 5f54c16ecf1944deb480c2b1da864060 P ddb27858eb984d79a4a766a8cec180d5: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260926 20:57:33.661414 12109 tablet_bootstrap.cc:492] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117: Bootstrap starting.
I20260926 20:57:33.661556 11814 heartbeater.cc:503] Master 127.0.98.62:41753 was elected leader, sending a full tablet report...
I20260926 20:57:33.662429 12109 tablet_bootstrap.cc:654] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:33.663010 12109 log.cc:824] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:33.664309 12109 tablet_bootstrap.cc:492] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117: No bootstrap required, opened a new log
I20260926 20:57:33.664381 12109 ts_tablet_manager.cc:1397] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:57:33.666360 12109 raft_consensus.cc:348] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0532a89e1723473b9a18462259472807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37175 } } peers { permanent_uuid: "ddb27858eb984d79a4a766a8cec180d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33189 } } peers { permanent_uuid: "26218a6553084295b115593e3bf5c117" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35269 } }
I20260926 20:57:33.666563 12109 raft_consensus.cc:374] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:33.666603 12109 raft_consensus.cc:729] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26218a6553084295b115593e3bf5c117, State: Initialized, Role: FOLLOWER
I20260926 20:57:33.666769 12109 consensus_queue.cc:260] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0532a89e1723473b9a18462259472807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37175 } } peers { permanent_uuid: "ddb27858eb984d79a4a766a8cec180d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33189 } } peers { permanent_uuid: "26218a6553084295b115593e3bf5c117" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35269 } }
I20260926 20:57:33.667063 12109 ts_tablet_manager.cc:1428] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:57:33.667273 12086 heartbeater.cc:503] Master 127.0.98.62:41753 was elected leader, sending a full tablet report...
Sep 26 20:57:33 dist-test-slave-fl7g krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260926 20:57:33.799211 11815 tablet.cc:2369] T 5f54c16ecf1944deb480c2b1da864060 P ddb27858eb984d79a4a766a8cec180d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:57:33.813221 11952 tablet.cc:2369] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:57:33.840628 12087 tablet.cc:2369] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:57:33.928315 12108 raft_consensus.cc:482] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:57:33.928489 12108 raft_consensus.cc:504] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0532a89e1723473b9a18462259472807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37175 } } peers { permanent_uuid: "ddb27858eb984d79a4a766a8cec180d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33189 } } peers { permanent_uuid: "26218a6553084295b115593e3bf5c117" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35269 } }
I20260926 20:57:33.928910 12108 leader_election.cc:288] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddb27858eb984d79a4a766a8cec180d5 (127.0.98.1:33189), 26218a6553084295b115593e3bf5c117 (127.0.98.3:35269)
I20260926 20:57:33.933226 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:33.929339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44583 (local address 127.0.98.3:35269)
0926 20:57:33.929386 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:33.929388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:33.929545 (+ 157us) server_negotiation.cc:407] Connection header received
0926 20:57:33.929644 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:33.929645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:33.929657 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:33.929684 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:33.929883 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.930627 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.932264 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.932830 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.932880 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.932892 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:33.933112 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 20:57:33.933154 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 20:57:33.933646 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:33.929150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35269 (local address 127.0.98.2:44583)
0926 20:57:33.929483 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:33.929493 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:33.929572 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:33.929744 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:33.929744 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:33.929746 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:33.929814 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:33.929816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.930774 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.930775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:33.932160 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:33.932162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.932952 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.932953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:33.933022 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.933026 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:57:33.933519 (+ 493us) client_negotiation.cc:240] Negotiation successful
0926 20:57:33.933567 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":26,"thread_start_us":65,"threads_started":1}
I20260926 20:57:33.933689 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:33.929324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40167 (local address 127.0.98.1:33189)
0926 20:57:33.929370 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:33.929372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:33.929493 (+ 121us) server_negotiation.cc:407] Connection header received
0926 20:57:33.929594 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:33.929595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:33.929609 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:33.929638 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:33.929846 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.930617 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.932268 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.932931 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.933180 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.933192 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:33.933447 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 20:57:33.933591 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:57:33.933935 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:33.929088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33189 (local address 127.0.98.2:40167)
0926 20:57:33.929422 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:33.929433 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:33.929520 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:33.929706 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:33.929706 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:33.929709 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:33.929770 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:33.929772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.930742 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.930742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:33.932141 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:33.932143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.933287 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.933288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:33.933364 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.933368 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:57:33.933805 (+ 437us) client_negotiation.cc:240] Negotiation successful
0926 20:57:33.933856 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":45,"thread_start_us":105,"threads_started":1}
I20260926 20:57:33.934163 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f54c16ecf1944deb480c2b1da864060" candidate_uuid: "0532a89e1723473b9a18462259472807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26218a6553084295b115593e3bf5c117" is_pre_election: true
I20260926 20:57:33.934379 12040 raft_consensus.cc:2454] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0532a89e1723473b9a18462259472807 in term 0.
I20260926 20:57:33.934505 11767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f54c16ecf1944deb480c2b1da864060" candidate_uuid: "0532a89e1723473b9a18462259472807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb27858eb984d79a4a766a8cec180d5" is_pre_election: true
I20260926 20:57:33.934651 11840 leader_election.cc:302] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0532a89e1723473b9a18462259472807, 26218a6553084295b115593e3bf5c117; no voters:
I20260926 20:57:33.934684 11767 raft_consensus.cc:2454] T 5f54c16ecf1944deb480c2b1da864060 P ddb27858eb984d79a4a766a8cec180d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0532a89e1723473b9a18462259472807 in term 0.
I20260926 20:57:33.934790 12108 raft_consensus.cc:2781] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:57:33.934842 12108 raft_consensus.cc:482] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:57:33.934983 12108 raft_consensus.cc:3037] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:33.936309 12108 raft_consensus.cc:504] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0532a89e1723473b9a18462259472807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37175 } } peers { permanent_uuid: "ddb27858eb984d79a4a766a8cec180d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33189 } } peers { permanent_uuid: "26218a6553084295b115593e3bf5c117" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35269 } }
I20260926 20:57:33.936491 12108 leader_election.cc:288] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [CANDIDATE]: Term 1 election: Requested vote from peers ddb27858eb984d79a4a766a8cec180d5 (127.0.98.1:33189), 26218a6553084295b115593e3bf5c117 (127.0.98.3:35269)
I20260926 20:57:33.936659 11767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f54c16ecf1944deb480c2b1da864060" candidate_uuid: "0532a89e1723473b9a18462259472807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb27858eb984d79a4a766a8cec180d5"
I20260926 20:57:33.936743 11767 raft_consensus.cc:3037] T 5f54c16ecf1944deb480c2b1da864060 P ddb27858eb984d79a4a766a8cec180d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:33.937625 11767 raft_consensus.cc:2454] T 5f54c16ecf1944deb480c2b1da864060 P ddb27858eb984d79a4a766a8cec180d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0532a89e1723473b9a18462259472807 in term 1.
I20260926 20:57:33.937837 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f54c16ecf1944deb480c2b1da864060" candidate_uuid: "0532a89e1723473b9a18462259472807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26218a6553084295b115593e3bf5c117"
I20260926 20:57:33.937871 11839 leader_election.cc:302] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0532a89e1723473b9a18462259472807, ddb27858eb984d79a4a766a8cec180d5; no voters:
I20260926 20:57:33.937920 12040 raft_consensus.cc:3037] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:33.937959 12108 raft_consensus.cc:2781] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:33.938109 12108 raft_consensus.cc:686] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [term 1 LEADER]: Becoming Leader. State: Replica: 0532a89e1723473b9a18462259472807, State: Running, Role: LEADER
I20260926 20:57:33.938226 12108 consensus_queue.cc:237] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0532a89e1723473b9a18462259472807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37175 } } peers { permanent_uuid: "ddb27858eb984d79a4a766a8cec180d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33189 } } peers { permanent_uuid: "26218a6553084295b115593e3bf5c117" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35269 } }
I20260926 20:57:33.938750 12040 raft_consensus.cc:2454] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0532a89e1723473b9a18462259472807 in term 1.
I20260926 20:57:33.939292 11601 catalog_manager.cc:5495] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0532a89e1723473b9a18462259472807 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0532a89e1723473b9a18462259472807" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0532a89e1723473b9a18462259472807" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb27858eb984d79a4a766a8cec180d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26218a6553084295b115593e3bf5c117" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35269 } health_report { overall_health: UNKNOWN } } }
I20260926 20:57:33.997817 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:33.994339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44350 (local address 127.0.98.2:37175)
0926 20:57:33.994465 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:33.994467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:33.994475 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:57:33.994514 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:33.994515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:33.994541 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:33.994610 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:33.994801 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.995513 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.996661 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.996951 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.997021 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.997230 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:33.997368 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:33.997446 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:33.997696 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 20:57:33.997719 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 20:57:34.000372 12040 raft_consensus.cc:1261] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117 [term 1 FOLLOWER]: Refusing update from remote peer 0532a89e1723473b9a18462259472807: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:57:34.000707 11767 raft_consensus.cc:1261] T 5f54c16ecf1944deb480c2b1da864060 P ddb27858eb984d79a4a766a8cec180d5 [term 1 FOLLOWER]: Refusing update from remote peer 0532a89e1723473b9a18462259472807: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:57:34.000787 12108 consensus_queue.cc:1035] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26218a6553084295b115593e3bf5c117" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:57:34.001091 12108 consensus_queue.cc:1035] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb27858eb984d79a4a766a8cec180d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:57:34.004130 12118 mvcc.cc:205] Tried to move back new op lower bound from 7333708816382050304 to 7333708816131596288. Current Snapshot: MvccSnapshot[applied={T|T < 7333708816382050304}]
I20260926 20:57:34.006117 12119 mvcc.cc:205] Tried to move back new op lower bound from 7333708816382050304 to 7333708816131596288. Current Snapshot: MvccSnapshot[applied={T|T < 7333708816382050304}]
I20260926 20:57:34.006227 12121 mvcc.cc:205] Tried to move back new op lower bound from 7333708816382050304 to 7333708816131596288. Current Snapshot: MvccSnapshot[applied={T|T < 7333708816382050304}]
I20260926 20:57:34.011348 11601 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:57:34.011472 11601 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:57:34.012527 11601 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 134b9796cf5f418fb72150f209d35a96: Sending DeleteTablet for 3 replicas of tablet 5f54c16ecf1944deb480c2b1da864060
I20260926 20:57:34.012893 11748 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f54c16ecf1944deb480c2b1da864060 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:57:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49046
I20260926 20:57:34.012909 12001 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f54c16ecf1944deb480c2b1da864060 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:57:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39284
I20260926 20:57:34.013798 11885 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f54c16ecf1944deb480c2b1da864060 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:57:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44340
I20260926 20:57:34.014554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 11683
I20260926 20:57:34.016450 12129 tablet_replica.cc:330] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807: stopping tablet replica
I20260926 20:57:34.016610 12129 raft_consensus.cc:2229] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:57:34.016847 12129 raft_consensus.cc:2258] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:57:34.017067 12128 tablet_replica.cc:330] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117: stopping tablet replica
I20260926 20:57:34.017558 12129 ts_tablet_manager.cc:1905] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:57:34.019470 12128 raft_consensus.cc:2229] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:57:34.019698 12128 raft_consensus.cc:2258] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:57:34.019964 12129 ts_tablet_manager.cc:1918] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:57:34.020128 12129 log.cc:1196] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/5f54c16ecf1944deb480c2b1da864060
I20260926 20:57:34.020573 12128 ts_tablet_manager.cc:1905] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:57:34.022293 12128 ts_tablet_manager.cc:1918] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:57:34.022675 12129 ts_tablet_manager.cc:1939] T 5f54c16ecf1944deb480c2b1da864060 P 0532a89e1723473b9a18462259472807: Deleting consensus metadata
I20260926 20:57:34.023170 11586 catalog_manager.cc:4841] TS 0532a89e1723473b9a18462259472807 (127.0.98.2:37175): tablet 5f54c16ecf1944deb480c2b1da864060 (table test-table [id=d8ffe5c6cdec40e4a055ea7b825669f6]) successfully deleted
I20260926 20:57:34.027336 12128 log.cc:1196] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/5f54c16ecf1944deb480c2b1da864060
I20260926 20:57:34.028137 12128 ts_tablet_manager.cc:1939] T 5f54c16ecf1944deb480c2b1da864060 P 26218a6553084295b115593e3bf5c117: Deleting consensus metadata
I20260926 20:57:34.028765 11588 catalog_manager.cc:4841] TS 26218a6553084295b115593e3bf5c117 (127.0.98.3:35269): tablet 5f54c16ecf1944deb480c2b1da864060 (table test-table [id=d8ffe5c6cdec40e4a055ea7b825669f6]) successfully deleted
W20260926 20:57:34.029072 11587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33189 (error 108)
I20260926 20:57:34.030128 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:34.029999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33189 (local address 127.0.0.1:49060)
0926 20:57:34.030050 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:34.030090 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 20:57:34.030390 11587 catalog_manager.cc:4584] TS ddb27858eb984d79a4a766a8cec180d5 (127.0.98.1:33189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f54c16ecf1944deb480c2b1da864060: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33189: connect: Connection refused (error 111)
I20260926 20:57:34.030681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 11820
I20260926 20:57:34.036707 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 11956
I20260926 20:57:34.042441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 11570
I20260926 20:57:34.047524 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 20:57:34.382814 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 20:57:34.382915 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 20:57:34.728152 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 20:57:34.728 UTC [11156] LOG: received smart shutdown request
2026-09-26 20:57:34.732 UTC [11156] LOG: background worker "logical replication launcher" (PID 11165) exited with exit code 1
2026-09-26 20:57:34.732 UTC [11160] LOG: shutting down
2026-09-26 20:57:34.939 UTC [11156] LOG: database system is shut down
2026-09-26T20:57:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47759 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:35 dist-test-slave-fl7g krb5kdc[12139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:35 dist-test-slave-fl7g krb5kdc[12139](info): set up 2 sockets
Sep 26 20:57:35 dist-test-slave-fl7g krb5kdc[12139](info): commencing operation
krb5kdc: starting...
W20260926 20:57:36.088326 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:57:36 dist-test-slave-fl7g krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 20:57:36.268210 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskVxazDW/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/postgres -l logfile start
2026-09-26 20:57:39.029 UTC [12169] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 20:57:39.030 UTC [12169] LOG: listening on IPv4 address "127.0.98.20", port 42473
2026-09-26 20:57:39.034 UTC [12169] LOG: listening on Unix socket "/tmp/.s.PGSQL.42473"
2026-09-26 20:57:39.057 UTC [12172] LOG: database system was shut down at 2026-09-26 20:57:37 UTC
2026-09-26 20:57:39.061 UTC [12169] LOG: database system is ready to accept connections
I20260926 20:57:40.028281 392 mini_postgres.cc:96] Postgres bound to 42473
2026-09-26 20:57:40.034 UTC [12180] FATAL: database "slave" does not exist
127.0.98.20:42473 - accepting connections
I20260926 20:57:40.034891 392 mini_ranger.cc:161] Starting Ranger...
I20260926 20:57:40.050057 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 20:57:40.647068 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 20:57:40.647156 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-admin
2026-09-26 20:57:40,947 [I] DB FLAVOR :POSTGRES
2026-09-26 20:57:40,948 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:57:40,948 [I] Checking connection..
2026-09-26 20:57:40,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 20:57:41,192 [I] Checking connection passed.
2026-09-26 20:57:41,192 [I] --------- Verifying version history table ---------
2026-09-26 20:57:41,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:57:41,439 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:57:41,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:57:41,714 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:57:41,714 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 20:57:41,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 20:57:41.978 UTC [12244] WARNING: there is no transaction in progress
2026-09-26 20:57:41,985 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 20:57:41,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:57:42,227 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 20:57:42,227 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:57:42,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 20:57:42,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 20:57:42,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fl7g.c.gcp-upstream.internal','N') ;"
2026-09-26 20:57:42,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:57:43,222 [I] Table x_portal_user does not exist in database ranger
2026-09-26 20:57:43,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:57:43,496 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 20:57:43,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:57:43,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:57:44,005 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 20:57:44,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 20:57:45.109 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 20:57:45.129 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 20:57:45.148 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 20:57:45.165 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 20:57:45.183 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 20:57:45.205 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 20:57:45.273 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 20:57:45.281 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 20:57:45.299 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 20:57:45.311 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 20:57:45.323 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 20:57:45.339 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 20:57:45.350 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 20:57:45.358 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 20:57:45.365 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 20:57:45.373 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 20:57:46,163 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 20:57:46,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 20:57:46,411 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 20:57:46,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:57:46,668 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 20:57:46,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:57:46,913 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 20:57:46,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:57:47,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:57:47,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fl7g.c.gcp-upstream.internal';"
2026-09-26 20:57:47,703 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 20:57:47,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 20:57:47,959 [I] DB_PATCHES have already been applied
I20260926 20:57:47.964695 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskVxazDW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 20:57:47.964759 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 20:57:47.968163 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:59763
Sep 26, 2026 8:57:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:57:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:57:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 8:57:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 8:57:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 8:57:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 8:57:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 20:57:49.090835 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.056s user 0.000s sys 0.009s
Sep 26, 2026 8:58:09 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 8:58:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 8:58:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 8:58:09 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 8:58:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 8:58:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 8:58:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 8:58:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 8:58:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 8:58:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 8:58:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 8:58:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 8:58:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 8:58:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 8:58:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 8:58:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 8:58:09 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 8:58:10 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 20:58:10.857115 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 20:58:10 dist-test-slave-fl7g krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456290, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 20:58:10.957695 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 20:58:10.967828 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 20:58:11.993189 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 20:58:11.993291 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-kms
I20260926 20:58:11.993317 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:42473
2026-09-26 20:58:12,158 [I] DB FLAVOR :POSTGRES
2026-09-26 20:58:12,158 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:58:12,158 [I] Checking connection
2026-09-26 20:58:12,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 20:58:12,393 [I] connection success
2026-09-26 20:58:12,393 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 20:58:12,393 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 20:58:12,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 20:58:12,640 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 20:58:12,640 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:58:12,641 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 20:58:12,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 20:58:12,914 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 20:58:13.389545 392 mini_ranger_kms.cc:310] Created kms service
I20260926 20:58:13.594311 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 20:58:13.646128 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 20:58:13 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456290, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:58:14.070994 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 20:58:14.071118 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskVxazDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskVxazDW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 20:58:14.071141 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 20:58:14.074568 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12532
I20260926 20:58:14.074661 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:34781
Sep 26, 2026 8:58:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:58:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:58:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 8:58:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 8:58:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 8:58:15 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 20:58:16.348193 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.391s user 0.002s sys 0.146s
I20260926 20:58:16.348284 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 20:58:16.348323 392 mini_ranger_kms.cc:401] 127.0.98.20:34781/kms/v1/keys
Sep 26 20:58:16 dist-test-slave-fl7g krb5kdc[12139](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790456296, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 20:58:16 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456296, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 20:58:17 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456296, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 20:58:19 dist-test-slave-fl7g krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:58:19.679972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45073
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45073
--ranger_config_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34781/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:58:19.767237 12583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:19.767549 12583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:19.767583 12583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:19.769052 12583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:58:19.769110 12583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:19.769127 12583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:19.769141 12583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:58:19.769179 12583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:58:19.771070 12583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:34781/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45073
--ranger_config_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:19.771377 12583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:19.771649 12583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:19.778380 12589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:19.778333 12591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:19.778606 12588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:19.778602 12583 server_base.cc:1034] running on GCE node
I20260926 20:58:19.778985 12583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:19.779330 12583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:19.780478 12583 hybrid_clock.cc:648] HybridClock initialized: now 1790456299780467 us; error 30 us; skew 500 ppm
Sep 26 20:58:19 dist-test-slave-fl7g krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:19.783438 12583 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:58:19.784005 12583 webserver.cc:460] Webserver started at http://127.0.98.62:36855/ using document root <none> and password file <none>
I20260926 20:58:19.784268 12583 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:19.784332 12583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:19.784516 12583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:58:19 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:58:19.832526 12583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "1da80bbb86c04ff788d339841a01bb3e"
format_stamp: "Formatted at 2026-09-26 20:58:19 on dist-test-slave-fl7g"
server_key: "1065c66db5b48a4d64e1498eea93a3b4"
server_key_iv: "879248cdd818bf7478a864bf73f2d1d5"
server_key_version: "kuduclusterkey@0"
I20260926 20:58:19.832994 12583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "1da80bbb86c04ff788d339841a01bb3e"
format_stamp: "Formatted at 2026-09-26 20:58:19 on dist-test-slave-fl7g"
server_key: "1065c66db5b48a4d64e1498eea93a3b4"
server_key_iv: "879248cdd818bf7478a864bf73f2d1d5"
server_key_version: "kuduclusterkey@0"
I20260926 20:58:19.834508 12583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:58:19.847110 12598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:19.847298 12583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:19.847386 12583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "1da80bbb86c04ff788d339841a01bb3e"
format_stamp: "Formatted at 2026-09-26 20:58:19 on dist-test-slave-fl7g"
server_key: "1065c66db5b48a4d64e1498eea93a3b4"
server_key_iv: "879248cdd818bf7478a864bf73f2d1d5"
server_key_version: "kuduclusterkey@0"
I20260926 20:58:19.847493 12583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:19.861603 12583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:19.863925 12583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:19.864091 12583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:19.869139 12583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45073
I20260926 20:58:19.869237 12650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45073 every 8 connection(s)
I20260926 20:58:19.869596 12583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:58:19.870428 12651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:19.872897 12651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e: Bootstrap starting.
I20260926 20:58:19.873517 12651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e: Neither blocks nor log segments found. Creating new log.
I20260926 20:58:19.873831 12651 log.cc:824] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:19.874584 12651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e: No bootstrap required, opened a new log
I20260926 20:58:19.876355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 12583
I20260926 20:58:19.876495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:58:19.876737 392 external_mini_cluster.cc:1428] Setting key 3a4fec479f9ea0674ecb63a4c0b9899e
I20260926 20:58:19.877056 12651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da80bbb86c04ff788d339841a01bb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45073 } }
I20260926 20:58:19.877559 12651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:19.877627 12651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da80bbb86c04ff788d339841a01bb3e, State: Initialized, Role: FOLLOWER
I20260926 20:58:19.877802 12651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da80bbb86c04ff788d339841a01bb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45073 } }
I20260926 20:58:19.877908 12651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:58:19.877964 12651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:58:19.878022 12651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:19.878894 12651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da80bbb86c04ff788d339841a01bb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45073 } }
I20260926 20:58:19.879017 12651 leader_election.cc:302] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1da80bbb86c04ff788d339841a01bb3e; no voters:
I20260926 20:58:19.879191 12651 leader_election.cc:288] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:58:19.879237 12656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:58:19.879441 12656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [term 1 LEADER]: Becoming Leader. State: Replica: 1da80bbb86c04ff788d339841a01bb3e, State: Running, Role: LEADER
I20260926 20:58:19.879559 12656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da80bbb86c04ff788d339841a01bb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45073 } }
I20260926 20:58:19.879590 12651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:58:19.879956 12656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1da80bbb86c04ff788d339841a01bb3e. Latest consensus state: current_term: 1 leader_uuid: "1da80bbb86c04ff788d339841a01bb3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da80bbb86c04ff788d339841a01bb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45073 } } }
I20260926 20:58:19.880017 12656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [sys.catalog]: This master's current role is: LEADER
I20260926 20:58:19.881192 12651 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:58:19.881649 12658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1da80bbb86c04ff788d339841a01bb3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da80bbb86c04ff788d339841a01bb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45073 } } }
I20260926 20:58:19.881737 12658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [sys.catalog]: This master's current role is: LEADER
Sep 26 20:58:19 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:19.884833 12661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:58:19.885818 12661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:58:19.891144 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:19.877325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34672 (local address 127.0.98.62:45073)
0926 20:58:19.877529 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:19.877541 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:19.877580 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:58:19.877711 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:19.877730 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:19.877793 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:19.877941 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:19.878834 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:19.880316 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:19.881502 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:19.882129 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:19.882249 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:19.885558 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:19.885586 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:19.885586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:19.885646 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:19.888686 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:19.889276 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:19.889279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:19.889279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:19.889410 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:19.889665 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:19.889665 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:19.889666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:19.890039 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:19.890642 (+ 603us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:19.890724 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 20:58:19.890888 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260926 20:58:19.892498 12661 catalog_manager.cc:1347] Generated new cluster ID: 673ffa2c30c94d80832952c525a93bfc
I20260926 20:58:19.892611 12661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:58:19.904650 12661 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:58:19.905503 12661 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:58:19.923581 12661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e: Generated new TSK 0
I20260926 20:58:19.923892 12661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:58:20.744590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34781/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45073
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:58:20.867451 12696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:20.867740 12696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:20.867784 12696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:20.869292 12696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:20.869352 12696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:20.869396 12696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:58:20.871744 12696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:34781/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:20.872022 12696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:20.872287 12696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:20.877830 12701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:20.877838 12704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:20.878028 12702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:20.878248 12696 server_base.cc:1034] running on GCE node
I20260926 20:58:20.878489 12696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:20.878758 12696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:20.879940 12696 hybrid_clock.cc:648] HybridClock initialized: now 1790456300879907 us; error 47 us; skew 500 ppm
Sep 26 20:58:20 dist-test-slave-fl7g krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:20.883571 12696 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:58:20.884194 12696 webserver.cc:460] Webserver started at http://127.0.98.1:34407/ using document root <none> and password file <none>
I20260926 20:58:20.884512 12696 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:20.884579 12696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:20.884773 12696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:58:20 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:58:20.901080 12696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "1c61b194d8f041c8892564467ff18aa6"
format_stamp: "Formatted at 2026-09-26 20:58:20 on dist-test-slave-fl7g"
server_key: "8dd3307cb26d7061a7e4ab8ad8613b90"
server_key_iv: "a7c394386307765b629c0c404982874a"
server_key_version: "kuduclusterkey@0"
I20260926 20:58:20.901641 12696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1c61b194d8f041c8892564467ff18aa6"
format_stamp: "Formatted at 2026-09-26 20:58:20 on dist-test-slave-fl7g"
server_key: "8dd3307cb26d7061a7e4ab8ad8613b90"
server_key_iv: "a7c394386307765b629c0c404982874a"
server_key_version: "kuduclusterkey@0"
I20260926 20:58:20.903688 12696 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:58:20.919013 12712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:20.919234 12696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:20.919333 12696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "1c61b194d8f041c8892564467ff18aa6"
format_stamp: "Formatted at 2026-09-26 20:58:20 on dist-test-slave-fl7g"
server_key: "8dd3307cb26d7061a7e4ab8ad8613b90"
server_key_iv: "a7c394386307765b629c0c404982874a"
server_key_version: "kuduclusterkey@0"
I20260926 20:58:20.919519 12696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:20.947364 12696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:20.954586 12696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:20.954780 12696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:20.955068 12696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:58:20.955494 12696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:58:20.955533 12696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:20.955555 12696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:58:20.955569 12696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:20.962852 12696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44357
I20260926 20:58:20.962896 12826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44357 every 8 connection(s)
I20260926 20:58:20.963284 12696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:58:20.963413 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 12696
I20260926 20:58:20.963517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:58:20.963768 392 external_mini_cluster.cc:1428] Setting key a7f91a5698475a4b8dce81a0f24b11ba
Sep 26 20:58:20 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 20:58:20.977543 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:20.964445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39365 (local address 127.0.98.62:45073)
0926 20:58:20.965817 (+ 1372us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:20.965819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:20.965827 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:58:20.965851 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:20.965852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:20.965865 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:20.965924 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:20.966605 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:20.967420 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:20.968625 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:20.968959 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:20.969021 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:20.971548 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:20.971550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:20.971550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:20.971558 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:20.975677 (+ 4119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:20.976271 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:20.976273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:20.976274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:20.976354 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:20.976696 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:20.976697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:20.976698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:20.976866 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:20.976956 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:20.977313 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 20:58:20.977424 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1347,"thread_start_us":58,"threads_started":1}
I20260926 20:58:20.977741 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:20.964432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45073 (local address 127.0.98.1:39365)
0926 20:58:20.964855 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:20.964899 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:20.965629 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:20.966067 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:20.966072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:20.966305 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:20.966503 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:20.966508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:20.967529 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:20.967530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:20.968536 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:20.968538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:20.969098 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:20.969099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:20.969263 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:20.969931 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:20.969943 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:20.971445 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:20.975818 (+ 4373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:20.975829 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:20.975832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:20.976171 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:20.976474 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:20.976475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:20.976475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:20.976613 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:20.976949 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:20.976954 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:20.977101 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 20:58:20.977263 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 20:58:20.977503 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":85,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 20:58:20.978794 12827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45073
I20260926 20:58:20.978901 12827 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:20.979130 12827 heartbeater.cc:511] Master 127.0.98.62:45073 requested a full tablet report, sending...
I20260926 20:58:20.979717 12615 ts_manager.cc:194] Registered new tserver with Master: 1c61b194d8f041c8892564467ff18aa6 (127.0.98.1:44357)
I20260926 20:58:20.980679 12615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:58:21.000061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34781/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45073
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:58:21.120872 12833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:21.121110 12833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:21.121152 12833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:21.123020 12833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:21.123088 12833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:21.123133 12833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:58:21.125530 12833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:34781/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:21.125850 12833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:21.126158 12833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:21.132920 12843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:21.132948 12841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:21.133313 12840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:21.134598 12833 server_base.cc:1034] running on GCE node
I20260926 20:58:21.134917 12833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
Sep 26 20:58:21 dist-test-slave-fl7g krb5kdc[12139](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790456301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:21.135272 12833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:21.136483 12833 hybrid_clock.cc:648] HybridClock initialized: now 1790456301136452 us; error 41 us; skew 500 ppm
Sep 26 20:58:21 dist-test-slave-fl7g krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:21.139839 12833 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:58:21.140475 12833 webserver.cc:460] Webserver started at http://127.0.98.2:33387/ using document root <none> and password file <none>
I20260926 20:58:21.140777 12833 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:21.140851 12833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:21.141053 12833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:58:21 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:58:21.155369 12833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "1cdc63b471d240c3bc1eb9e7522ac089"
format_stamp: "Formatted at 2026-09-26 20:58:21 on dist-test-slave-fl7g"
server_key: "15c07f8563fe446e9d3c1301eb538b6b"
server_key_iv: "d952a91884cd027bb8f0a7d245e055ce"
server_key_version: "kuduclusterkey@0"
I20260926 20:58:21.155856 12833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1cdc63b471d240c3bc1eb9e7522ac089"
format_stamp: "Formatted at 2026-09-26 20:58:21 on dist-test-slave-fl7g"
server_key: "15c07f8563fe446e9d3c1301eb538b6b"
server_key_iv: "d952a91884cd027bb8f0a7d245e055ce"
server_key_version: "kuduclusterkey@0"
I20260926 20:58:21.170588 12833 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.000s sys 0.002s
I20260926 20:58:21.194420 12850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:21.194737 12833 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260926 20:58:21.194850 12833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "1cdc63b471d240c3bc1eb9e7522ac089"
format_stamp: "Formatted at 2026-09-26 20:58:21 on dist-test-slave-fl7g"
server_key: "15c07f8563fe446e9d3c1301eb538b6b"
server_key_iv: "d952a91884cd027bb8f0a7d245e055ce"
server_key_version: "kuduclusterkey@0"
I20260926 20:58:21.194962 12833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:21.216814 12833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:21.219437 12833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:21.219667 12833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:21.220010 12833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:58:21.220454 12833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:58:21.220548 12833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:21.220616 12833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:58:21.220674 12833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:21.227296 12833 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45135
I20260926 20:58:21.227818 12833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:58:21.231890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 12833
I20260926 20:58:21.232012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:58:21.232227 392 external_mini_cluster.cc:1428] Setting key 3fea55af49d46e44b716392bc179a141
I20260926 20:58:21.233043 12964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45135 every 8 connection(s)
Sep 26 20:58:21 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:21.244177 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:21.230568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52043 (local address 127.0.98.62:45073)
0926 20:58:21.230616 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:21.230618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:21.230993 (+ 375us) server_negotiation.cc:407] Connection header received
0926 20:58:21.231708 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:21.231709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:21.231734 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:21.231770 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:21.234984 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:21.235795 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:21.237402 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:21.237747 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:21.237792 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:21.240527 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:21.240528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:21.240529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:21.240537 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:21.242493 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:21.243018 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:21.243019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:21.243019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:21.243079 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:21.243356 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:21.243356 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:21.243357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:21.243517 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:21.243568 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:21.243957 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 20:58:21.244050 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:58:21.245028 12967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:21.230273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45073 (local address 127.0.98.2:52043)
0926 20:58:21.230858 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:21.230905 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:21.231571 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:21.232114 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:21.232119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:21.234553 (+ 2434us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:21.234861 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:21.234868 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:21.235918 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:21.235919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:21.237296 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:21.237299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:21.237899 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:21.237899 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:21.238045 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:21.238692 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:21.238703 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:21.240389 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:21.242596 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:21.242604 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:21.242606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:21.242926 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:21.243143 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:21.243143 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:21.243144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:21.243283 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:21.243601 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:21.243608 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:21.243802 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 20:58:21.244599 (+ 797us) client_negotiation.cc:240] Negotiation successful
0926 20:58:21.244785 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":94,"threads_started":1}
I20260926 20:58:21.245896 12965 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45073
I20260926 20:58:21.245994 12965 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:21.246187 12965 heartbeater.cc:511] Master 127.0.98.62:45073 requested a full tablet report, sending...
I20260926 20:58:21.246587 12615 ts_manager.cc:194] Registered new tserver with Master: 1cdc63b471d240c3bc1eb9e7522ac089 (127.0.98.2:45135)
I20260926 20:58:21.247116 12615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52043
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:58:21.283826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34781/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45073
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:58:21.405122 12970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:21.405536 12970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:21.405637 12970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:21.407570 12970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:21.407821 12970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:21.407920 12970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:58:21.410357 12970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:34781/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:21.410893 12970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:21.411231 12970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:21.418215 12976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:21.418648 12978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:21.419909 12970 server_base.cc:1034] running on GCE node
W20260926 20:58:21.420966 12975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:21.421378 12970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:21.421697 12970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:21.424943 12970 hybrid_clock.cc:648] HybridClock initialized: now 1790456301424324 us; error 635 us; skew 500 ppm
Sep 26 20:58:21 dist-test-slave-fl7g krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:21.428052 12970 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:58:21.428658 12970 webserver.cc:460] Webserver started at http://127.0.98.3:42901/ using document root <none> and password file <none>
I20260926 20:58:21.428911 12970 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:21.428964 12970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:21.429091 12970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:58:21 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:58:21.442292 12970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "290922525e6a4b078aed0d7013736c15"
format_stamp: "Formatted at 2026-09-26 20:58:21 on dist-test-slave-fl7g"
server_key: "09447ea0afbe4aea885cbff99d0e8e59"
server_key_iv: "ce23c5683f3d4bff3b287f9d9574c01d"
server_key_version: "kuduclusterkey@0"
I20260926 20:58:21.442740 12970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "290922525e6a4b078aed0d7013736c15"
format_stamp: "Formatted at 2026-09-26 20:58:21 on dist-test-slave-fl7g"
server_key: "09447ea0afbe4aea885cbff99d0e8e59"
server_key_iv: "ce23c5683f3d4bff3b287f9d9574c01d"
server_key_version: "kuduclusterkey@0"
I20260926 20:58:21.448267 12970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260926 20:58:21.467298 12985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:21.467581 12970 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.000s
I20260926 20:58:21.467674 12970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "290922525e6a4b078aed0d7013736c15"
format_stamp: "Formatted at 2026-09-26 20:58:21 on dist-test-slave-fl7g"
server_key: "09447ea0afbe4aea885cbff99d0e8e59"
server_key_iv: "ce23c5683f3d4bff3b287f9d9574c01d"
server_key_version: "kuduclusterkey@0"
I20260926 20:58:21.467772 12970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:21.486689 12970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:21.489425 12970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:21.489665 12970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:21.489987 12970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:58:21.490430 12970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:58:21.490509 12970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:21.490583 12970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:58:21.490648 12970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:21.498695 12970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40391
I20260926 20:58:21.499261 12970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:58:21.501195 13098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40391 every 8 connection(s)
I20260926 20:58:21.507874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 12970
I20260926 20:58:21.507989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:58:21.508203 392 external_mini_cluster.cc:1428] Setting key 236e548a859460c0a27695d3b724a473
Sep 26 20:58:21 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:21.518671 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:21.501917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45215 (local address 127.0.98.62:45073)
0926 20:58:21.501965 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:21.501968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:21.502493 (+ 525us) server_negotiation.cc:407] Connection header received
0926 20:58:21.503237 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:21.503238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:21.503253 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:21.503286 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:21.504171 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:21.504957 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:21.506479 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:21.506768 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:21.506816 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:21.511551 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:21.511553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:21.511553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:21.511563 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:21.513645 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:21.517234 (+ 3589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:21.517236 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:21.517236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:21.517322 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:21.517762 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:21.517763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:21.517763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:21.517927 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:21.517975 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:21.518448 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 20:58:21.518565 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:58:21.520372 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:21.501907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45073 (local address 127.0.98.3:45215)
0926 20:58:21.502355 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:21.502401 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:21.503102 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:21.503484 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:21.503489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:21.503754 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:21.504051 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:21.504059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:21.505090 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:21.505091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:21.506339 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:21.506342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:21.506927 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:21.506928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:21.507089 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:21.509513 (+ 2424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:21.509526 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:21.511398 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:21.516543 (+ 5145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:21.516558 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:21.516561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:21.516987 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:21.517418 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:21.517419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:21.517419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:21.517657 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:21.518029 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:21.518036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:21.518255 (+ 219us) client_negotiation.cc:769] Sending connection context
0926 20:58:21.519095 (+ 840us) client_negotiation.cc:240] Negotiation successful
0926 20:58:21.519295 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":83,"threads_started":1}
I20260926 20:58:21.521202 13099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45073
I20260926 20:58:21.521317 13099 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:21.521550 13099 heartbeater.cc:511] Master 127.0.98.62:45073 requested a full tablet report, sending...
I20260926 20:58:21.521996 12615 ts_manager.cc:194] Registered new tserver with Master: 290922525e6a4b078aed0d7013736c15 (127.0.98.3:40391)
I20260926 20:58:21.522466 12615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45215
I20260926 20:58:21.531049 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:58:21.531836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 12696
I20260926 20:58:21.537696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 12833
I20260926 20:58:21.543678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 12970
I20260926 20:58:21.550561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 12583
I20260926 20:58:21.558398 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45073
--webserver_interface=127.0.98.62
--webserver_port=36855
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45073
--ranger_config_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34781/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34781/kms with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:58:21.656232 13102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:21.656569 13102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:21.656598 13102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:21.657878 13102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:58:21.657935 13102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:21.657950 13102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:21.657963 13102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:58:21.657975 13102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:58:21.659705 13102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34781/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45073
--ranger_config_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:21.659976 13102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:21.660240 13102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:21.666530 13107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:21.666540 13108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:21.666594 13110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:21.666841 13102 server_base.cc:1034] running on GCE node
I20260926 20:58:21.667106 13102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:21.667443 13102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:21.668668 13102 hybrid_clock.cc:648] HybridClock initialized: now 1790456301668641 us; error 41 us; skew 500 ppm
Sep 26 20:58:21 dist-test-slave-fl7g krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:21.671566 13102 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:58:21.672155 13102 webserver.cc:460] Webserver started at http://127.0.98.62:36855/ using document root <none> and password file <none>
I20260926 20:58:21.672443 13102 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:21.672519 13102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:58:21 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:58:21.727186 13102 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:21.727946 13118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:21.728171 13102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:21.728287 13102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "1da80bbb86c04ff788d339841a01bb3e"
format_stamp: "Formatted at 2026-09-26 20:58:19 on dist-test-slave-fl7g"
server_key: "1065c66db5b48a4d64e1498eea93a3b4"
server_key_iv: "879248cdd818bf7478a864bf73f2d1d5"
server_key_version: "kuduclusterkey@0"
I20260926 20:58:21.728789 13102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:21.759420 13102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:21.761288 13102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:21.761408 13102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:21.765709 13102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45073
I20260926 20:58:21.765748 13170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45073 every 8 connection(s)
I20260926 20:58:21.766120 13102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:58:21.767354 13171 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:58:21.768042 13171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e: Bootstrap starting.
I20260926 20:58:21.775875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 13102
I20260926 20:58:21.776077 13171 log.cc:824] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:21.776203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34781/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44357
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45073
--webserver_port=34407
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34781/kms with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:58:21.778246 13171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:58:21.778550 13171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e: Bootstrap complete.
I20260926 20:58:21.781061 13171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da80bbb86c04ff788d339841a01bb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45073 } }
I20260926 20:58:21.781301 13171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da80bbb86c04ff788d339841a01bb3e, State: Initialized, Role: FOLLOWER
I20260926 20:58:21.781469 13171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da80bbb86c04ff788d339841a01bb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45073 } }
I20260926 20:58:21.781539 13171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:58:21.781569 13171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:58:21.781644 13171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:58:21.782687 13171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da80bbb86c04ff788d339841a01bb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45073 } }
I20260926 20:58:21.782778 13171 leader_election.cc:302] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1da80bbb86c04ff788d339841a01bb3e; no voters:
I20260926 20:58:21.782999 13171 leader_election.cc:288] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:58:21.783044 13175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:58:21.783195 13175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [term 2 LEADER]: Becoming Leader. State: Replica: 1da80bbb86c04ff788d339841a01bb3e, State: Running, Role: LEADER
I20260926 20:58:21.783289 13171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:58:21.783345 13175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da80bbb86c04ff788d339841a01bb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45073 } }
I20260926 20:58:21.783715 13176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1da80bbb86c04ff788d339841a01bb3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da80bbb86c04ff788d339841a01bb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45073 } } }
I20260926 20:58:21.783806 13176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [sys.catalog]: This master's current role is: LEADER
I20260926 20:58:21.784019 13177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1da80bbb86c04ff788d339841a01bb3e. Latest consensus state: current_term: 2 leader_uuid: "1da80bbb86c04ff788d339841a01bb3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da80bbb86c04ff788d339841a01bb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45073 } } }
I20260926 20:58:21.784092 13177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e [sys.catalog]: This master's current role is: LEADER
I20260926 20:58:21.784211 13182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:58:21.784773 13182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:58:21.784960 13182 catalog_manager.cc:1259] Loaded cluster ID: 673ffa2c30c94d80832952c525a93bfc
I20260926 20:58:21.785033 13182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:58:21.785142 13171 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:58:21.786273 13182 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:58:21.786460 13182 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e: Loaded TSK: 0
I20260926 20:58:21.786693 13182 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:58:21.892870 13174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:21.893263 13174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:21.893361 13174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:21.895328 13174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:21.895459 13174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:21.895545 13174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:58:21.898012 13174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34781/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34407
--tserver_master_addrs=127.0.98.62:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:21.898598 13174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:21.898913 13174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:21.905652 13209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:21.905887 13208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:21.905659 13211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:21.906598 13174 server_base.cc:1034] running on GCE node
I20260926 20:58:21.906821 13174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:21.907120 13174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:21.908303 13174 hybrid_clock.cc:648] HybridClock initialized: now 1790456301908270 us; error 47 us; skew 500 ppm
Sep 26 20:58:21 dist-test-slave-fl7g krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:21.911675 13174 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:58:21.912204 13174 webserver.cc:460] Webserver started at http://127.0.98.1:34407/ using document root <none> and password file <none>
I20260926 20:58:21.912495 13174 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:21.912566 13174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:58:21 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:58:21.939970 13174 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:58:21.940960 13219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:21.941177 13174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:58:21.941308 13174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "1c61b194d8f041c8892564467ff18aa6"
format_stamp: "Formatted at 2026-09-26 20:58:20 on dist-test-slave-fl7g"
server_key: "8dd3307cb26d7061a7e4ab8ad8613b90"
server_key_iv: "a7c394386307765b629c0c404982874a"
server_key_version: "kuduclusterkey@0"
I20260926 20:58:21.941871 13174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:21.966724 13174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:21.969187 13174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:21.969337 13174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:21.969587 13174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:58:21.969921 13174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:58:21.969976 13174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:21.970021 13174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:58:21.970047 13174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:21.975850 13174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44357
I20260926 20:58:21.976006 13332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44357 every 8 connection(s)
I20260926 20:58:21.976331 13174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:58:21.977658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 13174
I20260926 20:58:21.977931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34781/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45135
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45073
--webserver_port=33387
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34781/kms with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:58:21 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:21.993479 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:21.977099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45073 (local address 127.0.98.1:47441)
0926 20:58:21.977286 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:21.977335 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:21.977948 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:21.978646 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:21.978650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:21.978907 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:21.979216 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:21.979223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:21.980970 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:21.980971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:21.982387 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:21.982391 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:21.983520 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:21.983521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:21.983685 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:21.984386 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:21.984398 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:21.987532 (+ 3134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:21.991328 (+ 3796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:21.991337 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:21.991341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:21.991681 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:21.991985 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:21.991986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:21.991987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:21.992155 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:21.992711 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:21.992718 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:21.992901 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 20:58:21.993019 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 20:58:21.993209 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":107,"threads_started":1}
I20260926 20:58:21.994164 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:21.977243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47441 (local address 127.0.98.62:45073)
0926 20:58:21.977431 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:21.977439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:21.977515 (+ 76us) server_negotiation.cc:407] Connection header received
0926 20:58:21.978265 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:21.978274 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:21.978334 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:21.978464 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:21.979349 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:21.980841 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:21.982493 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:21.983267 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:21.983394 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:21.987705 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:21.987711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:21.987712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:21.987727 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:21.991208 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:21.991787 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:21.991791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:21.991791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:21.991905 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:21.992234 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:21.992235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:21.992235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:21.992590 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:21.992679 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:21.993797 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0926 20:58:21.993942 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260926 20:58:21.994930 13333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45073
I20260926 20:58:21.995040 13333 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:21.995258 13333 heartbeater.cc:511] Master 127.0.98.62:45073 requested a full tablet report, sending...
W20260926 20:58:21.996134 13333 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45073 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 20:58:21.996217 13333 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:21.996307 13333 heartbeater.cc:511] Master 127.0.98.62:45073 requested a full tablet report, sending...
I20260926 20:58:21.996681 13333 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:21.996763 13333 heartbeater.cc:511] Master 127.0.98.62:45073 requested a full tablet report, sending...
W20260926 20:58:21.997067 13333 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:58:22.091933 13337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:22.092244 13337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:22.092296 13337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:22.094143 13337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:22.094209 13337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:22.094260 13337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:58:22.096727 13337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34781/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33387
--tserver_master_addrs=127.0.98.62:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:22.097014 13337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:22.097309 13337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:22.103107 13342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:22.103242 13345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:22.103107 13343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:22.103628 13337 server_base.cc:1034] running on GCE node
I20260926 20:58:22.103962 13337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:22.104377 13337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:22.105710 13337 hybrid_clock.cc:648] HybridClock initialized: now 1790456302105689 us; error 27 us; skew 500 ppm
Sep 26 20:58:22 dist-test-slave-fl7g krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:22.109563 13337 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:58:22.110246 13337 webserver.cc:460] Webserver started at http://127.0.98.2:33387/ using document root <none> and password file <none>
I20260926 20:58:22.110556 13337 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:22.110661 13337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:58:22 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:58:22.130235 13337 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:22.131278 13353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:22.131529 13337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:22.131636 13337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "1cdc63b471d240c3bc1eb9e7522ac089"
format_stamp: "Formatted at 2026-09-26 20:58:21 on dist-test-slave-fl7g"
server_key: "15c07f8563fe446e9d3c1301eb538b6b"
server_key_iv: "d952a91884cd027bb8f0a7d245e055ce"
server_key_version: "kuduclusterkey@0"
I20260926 20:58:22.132153 13337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:22.161484 13337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:22.165349 13337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:22.165532 13337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:22.165832 13337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:58:22.166195 13337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:58:22.166252 13337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:22.166296 13337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:58:22.166323 13337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:22.172230 13337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45135
I20260926 20:58:22.172464 13466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45135 every 8 connection(s)
I20260926 20:58:22.172756 13337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:58:22.175081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 13337
I20260926 20:58:22.175413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34781/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40391
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45073
--webserver_port=42901
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34781/kms with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:58:22 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:22.185442 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:22.173599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53931 (local address 127.0.98.62:45073)
0926 20:58:22.173677 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:22.173680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:22.173897 (+ 217us) server_negotiation.cc:407] Connection header received
0926 20:58:22.174551 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:22.174553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:22.174571 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:22.174665 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:22.175441 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:22.176052 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:22.177620 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:22.177986 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:22.178091 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:22.181190 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:22.181192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:22.181192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:22.181203 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:22.183337 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:22.183995 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:22.183996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:22.183997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:22.184081 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:22.184507 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:22.184508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:22.184509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:22.184679 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:22.184770 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:22.185174 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 20:58:22.185315 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:58:22.185722 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:22.173570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45073 (local address 127.0.98.2:53931)
0926 20:58:22.173754 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:22.173801 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:22.174420 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:22.174768 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:22.174773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:22.175048 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:22.175334 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:22.175340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:22.176172 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:22.176173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:22.177495 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:22.177499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:22.178129 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:22.178131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:22.178288 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:22.179040 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:22.179052 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:22.181052 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:22.183466 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:22.183476 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:22.183479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:22.183898 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:22.184170 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:22.184171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:22.184171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:22.184440 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:22.184766 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:22.184773 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:22.184979 (+ 206us) client_negotiation.cc:769] Sending connection context
0926 20:58:22.185094 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:58:22.185284 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":113,"threads_started":1}
I20260926 20:58:22.186302 13467 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45073
I20260926 20:58:22.186417 13467 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:22.186648 13467 heartbeater.cc:511] Master 127.0.98.62:45073 requested a full tablet report, sending...
W20260926 20:58:22.187454 13467 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45073 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 20:58:22.187529 13467 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:22.187595 13467 heartbeater.cc:511] Master 127.0.98.62:45073 requested a full tablet report, sending...
I20260926 20:58:22.187992 13467 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:22.188092 13467 heartbeater.cc:511] Master 127.0.98.62:45073 requested a full tablet report, sending...
W20260926 20:58:22.188474 13467 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:58:22.284580 13470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:22.284895 13470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:22.284946 13470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:22.287115 13470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:22.287189 13470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:22.287233 13470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:58:22.289834 13470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34781/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42901
--tserver_master_addrs=127.0.98.62:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:22.290129 13470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:22.290400 13470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:22.297072 13478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:22.297072 13476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:22.297582 13475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:22.297938 13470 server_base.cc:1034] running on GCE node
I20260926 20:58:22.298213 13470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:22.298517 13470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:22.299705 13470 hybrid_clock.cc:648] HybridClock initialized: now 1790456302299695 us; error 34 us; skew 500 ppm
Sep 26 20:58:22 dist-test-slave-fl7g krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:22.302479 13470 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:58:22.303047 13470 webserver.cc:460] Webserver started at http://127.0.98.3:42901/ using document root <none> and password file <none>
I20260926 20:58:22.303339 13470 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:22.303429 13470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:58:22 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:58:22.320667 13470 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:58:22.322006 13487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:22.322198 13470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:22.322295 13470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "290922525e6a4b078aed0d7013736c15"
format_stamp: "Formatted at 2026-09-26 20:58:21 on dist-test-slave-fl7g"
server_key: "09447ea0afbe4aea885cbff99d0e8e59"
server_key_iv: "ce23c5683f3d4bff3b287f9d9574c01d"
server_key_version: "kuduclusterkey@0"
I20260926 20:58:22.322758 13470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:22.355088 13470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:22.358263 13470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:22.358455 13470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:22.358733 13470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:58:22.359069 13470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:58:22.359125 13470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:22.359169 13470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:58:22.359196 13470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:22.364874 13470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40391
I20260926 20:58:22.365381 13470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:58:22.366341 13600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40391 every 8 connection(s)
I20260926 20:58:22.373487 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 13470
Sep 26 20:58:22 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:22.381105 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:22.367638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45073 (local address 127.0.98.3:44229)
0926 20:58:22.367936 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:22.367986 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:22.368789 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:22.369130 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:22.369135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:22.369442 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:22.369778 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:22.369785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:22.370969 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:22.370970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:22.372482 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:22.372486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:22.373107 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:22.373108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:22.373280 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:22.373930 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:22.373943 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:22.375742 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:22.377875 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:22.377886 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:22.377893 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:22.378269 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:22.378513 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:22.378514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:22.378514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:22.378675 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:22.380237 (+ 1562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:22.380246 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:22.380507 (+ 261us) client_negotiation.cc:769] Sending connection context
0926 20:58:22.380631 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 20:58:22.380819 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":100,"threads_started":1}
I20260926 20:58:22.381901 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:22.367578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44229 (local address 127.0.98.62:45073)
0926 20:58:22.367631 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:22.367634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:22.368108 (+ 474us) server_negotiation.cc:407] Connection header received
0926 20:58:22.368931 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:22.368933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:22.368947 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:22.369025 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:22.369943 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:22.370807 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:22.372620 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:22.372948 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:22.373031 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:22.375871 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:22.375873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:22.375873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:22.375881 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:22.377639 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:22.378365 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:22.378366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:22.378366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:22.378432 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:22.379948 (+ 1516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:22.379949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:22.379950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:22.380128 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:22.380205 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:22.381690 (+ 1485us) server_negotiation.cc:299] Negotiation successful
0926 20:58:22.381795 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:58:22.382546 13601 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45073
I20260926 20:58:22.382712 13601 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:22.383059 13601 heartbeater.cc:511] Master 127.0.98.62:45073 requested a full tablet report, sending...
W20260926 20:58:22.383780 13601 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45073 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 20:58:22.383920 13601 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:22.384116 13601 heartbeater.cc:511] Master 127.0.98.62:45073 requested a full tablet report, sending...
I20260926 20:58:22.384332 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:22.374407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34682 (local address 127.0.98.62:45073)
0926 20:58:22.375013 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:22.375015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:22.375025 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:58:22.375053 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:22.375053 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:22.375068 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:22.376592 (+ 1524us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:22.376639 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:22.377617 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:22.378820 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:22.379134 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:22.379199 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:22.380784 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:22.380786 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:22.380786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:22.380793 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:22.382615 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:22.383023 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:22.383024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:22.383024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:22.383086 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:22.383320 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:22.383321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:22.383321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:22.383458 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:22.383637 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:22.384111 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 20:58:22.384198 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":49,"threads_started":1}
I20260926 20:58:22.384944 13601 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:22.385406 13601 heartbeater.cc:511] Master 127.0.98.62:45073 requested a full tablet report, sending...
W20260926 20:58:22.385831 13601 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 20:58:22 dist-test-slave-fl7g krb5kdc[12139](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790456302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:22.997268 13333 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:22.997440 13333 heartbeater.cc:511] Master 127.0.98.62:45073 requested a full tablet report, sending...
I20260926 20:58:22.997960 13135 ts_manager.cc:194] Registered new tserver with Master: 1c61b194d8f041c8892564467ff18aa6 (127.0.98.1:44357)
I20260926 20:58:22.998973 13135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47441
I20260926 20:58:23.188680 13467 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:23.188862 13467 heartbeater.cc:511] Master 127.0.98.62:45073 requested a full tablet report, sending...
I20260926 20:58:23.189263 13135 ts_manager.cc:194] Registered new tserver with Master: 1cdc63b471d240c3bc1eb9e7522ac089 (127.0.98.2:45135)
I20260926 20:58:23.189822 13135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53931
Sep 26 20:58:23 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:58:23.386024 13601 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:23.386194 13601 heartbeater.cc:511] Master 127.0.98.62:45073 requested a full tablet report, sending...
I20260926 20:58:23.386548 13135 ts_manager.cc:194] Registered new tserver with Master: 290922525e6a4b078aed0d7013736c15 (127.0.98.3:40391)
I20260926 20:58:23.386988 13135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44229
I20260926 20:58:23.394680 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:58:23.404354 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:23.395842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34686 (local address 127.0.98.62:45073)
0926 20:58:23.396112 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:23.396114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:23.396322 (+ 208us) server_negotiation.cc:407] Connection header received
0926 20:58:23.396348 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:23.396350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:23.396361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:23.396447 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:23.397193 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.397977 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.399001 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.399320 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.399390 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:23.401007 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:23.401009 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:23.401009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:23.401017 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:23.402815 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:23.403199 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:23.403200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:23.403201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:23.403271 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:23.403482 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:23.403483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:23.403483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:23.403610 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:23.403693 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:23.404147 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 20:58:23.404247 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":65,"threads_started":1}
I20260926 20:58:23.406929 13135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:58:23.409756 13135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:58:23.426600 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:23.419531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44357 (local address 127.0.0.1:59890)
0926 20:58:23.420983 (+ 1452us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:23.421027 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:23.421090 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:23.421607 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:23.421610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:23.421612 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:23.421693 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:23.421696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.423314 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.423316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:23.425154 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:23.425157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.426276 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.426277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:23.426358 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:23.426374 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 20:58:23.426447 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 20:58:23.426507 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1421,"thread_start_us":71,"threads_started":1}
I20260926 20:58:23.427296 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:23.420024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.0.98.1:44357)
0926 20:58:23.421232 (+ 1208us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:23.421237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:23.421248 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:58:23.421284 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:23.421286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:23.421301 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:23.421373 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:23.421783 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.423205 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.425256 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.426078 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.426147 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:23.426240 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:23.427146 (+ 906us) server_negotiation.cc:299] Negotiation successful
0926 20:58:23.427198 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1162,"thread_start_us":57,"threads_started":1}
I20260926 20:58:23.428236 13267 tablet_service.cc:1467] Processing CreateTablet for tablet 29569e04b5e24d0ea5e5a2ace043bd2f (DEFAULT_TABLE table=test-table [id=67c455d91e1242ecb793ce5bb1439403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:58:23.428642 13267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29569e04b5e24d0ea5e5a2ace043bd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:23.435660 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:23.419791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40391 (local address 127.0.0.1:57474)
0926 20:58:23.421482 (+ 1691us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:23.421492 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:23.421538 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:23.425541 (+ 4003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:23.425543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:23.425545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:23.425586 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:23.425587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.433027 (+ 7440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.433029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:23.434258 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:23.434262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.435394 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.435395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:23.435464 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:23.435469 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:58:23.435523 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:58:23.435564 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1672,"thread_start_us":73,"threads_started":1}
I20260926 20:58:23.436312 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:23.419892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57474 (local address 127.0.98.3:40391)
0926 20:58:23.425306 (+ 5414us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:23.425312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:23.425322 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:58:23.425357 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:23.425359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:23.425376 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:23.425453 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:23.425656 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.432916 (+ 7260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.434375 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.435193 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.435272 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:23.435355 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:23.436152 (+ 797us) server_negotiation.cc:299] Negotiation successful
0926 20:58:23.436208 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5372,"thread_start_us":41,"threads_started":1}
I20260926 20:58:23.436333 13635 tablet_bootstrap.cc:492] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1c61b194d8f041c8892564467ff18aa6: Bootstrap starting.
I20260926 20:58:23.436645 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:23.420330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45135 (local address 127.0.0.1:46182)
0926 20:58:23.425124 (+ 4794us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:23.425136 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:23.425186 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:23.432831 (+ 7645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:23.432833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:23.432837 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:23.432896 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:23.432899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.434229 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.434230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:23.435381 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:23.435383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.436382 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.436383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:23.436483 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:23.436489 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:58:23.436517 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 20:58:23.436560 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4774,"thread_start_us":64,"threads_started":1}
I20260926 20:58:23.437278 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:23.420620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.0.98.2:45135)
0926 20:58:23.432531 (+ 11911us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:23.432537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:23.432552 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:58:23.432594 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:23.432597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:23.432616 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:23.432728 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:23.432977 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.434109 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.435496 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.436279 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.437002 (+ 723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:23.437125 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:23.437154 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 20:58:23.437202 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11861,"thread_start_us":65,"threads_started":1}
I20260926 20:58:23.437280 13635 tablet_bootstrap.cc:654] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1c61b194d8f041c8892564467ff18aa6: Neither blocks nor log segments found. Creating new log.
I20260926 20:58:23.437276 13535 tablet_service.cc:1467] Processing CreateTablet for tablet 29569e04b5e24d0ea5e5a2ace043bd2f (DEFAULT_TABLE table=test-table [id=67c455d91e1242ecb793ce5bb1439403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:58:23.437608 13635 log.cc:824] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1c61b194d8f041c8892564467ff18aa6: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:23.437640 13535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29569e04b5e24d0ea5e5a2ace043bd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:23.438056 13401 tablet_service.cc:1467] Processing CreateTablet for tablet 29569e04b5e24d0ea5e5a2ace043bd2f (DEFAULT_TABLE table=test-table [id=67c455d91e1242ecb793ce5bb1439403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:58:23.438366 13401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29569e04b5e24d0ea5e5a2ace043bd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:23.442215 13638 tablet_bootstrap.cc:492] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089: Bootstrap starting.
I20260926 20:58:23.442940 13638 tablet_bootstrap.cc:654] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089: Neither blocks nor log segments found. Creating new log.
I20260926 20:58:23.443265 13638 log.cc:824] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:23.444610 13637 tablet_bootstrap.cc:492] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15: Bootstrap starting.
I20260926 20:58:23.445394 13637 tablet_bootstrap.cc:654] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15: Neither blocks nor log segments found. Creating new log.
I20260926 20:58:23.445789 13637 log.cc:824] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:23.446195 13635 tablet_bootstrap.cc:492] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1c61b194d8f041c8892564467ff18aa6: No bootstrap required, opened a new log
I20260926 20:58:23.446277 13635 ts_tablet_manager.cc:1397] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1c61b194d8f041c8892564467ff18aa6: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260926 20:58:23.446786 13638 tablet_bootstrap.cc:492] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089: No bootstrap required, opened a new log
I20260926 20:58:23.446851 13638 ts_tablet_manager.cc:1397] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 20:58:23.449527 13638 raft_consensus.cc:348] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdc63b471d240c3bc1eb9e7522ac089" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "1c61b194d8f041c8892564467ff18aa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44357 } } peers { permanent_uuid: "290922525e6a4b078aed0d7013736c15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40391 } }
I20260926 20:58:23.449539 13635 raft_consensus.cc:348] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1c61b194d8f041c8892564467ff18aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdc63b471d240c3bc1eb9e7522ac089" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "1c61b194d8f041c8892564467ff18aa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44357 } } peers { permanent_uuid: "290922525e6a4b078aed0d7013736c15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40391 } }
I20260926 20:58:23.449743 13638 raft_consensus.cc:374] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:23.449792 13638 raft_consensus.cc:729] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cdc63b471d240c3bc1eb9e7522ac089, State: Initialized, Role: FOLLOWER
I20260926 20:58:23.449916 13635 raft_consensus.cc:374] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1c61b194d8f041c8892564467ff18aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:23.449940 13638 consensus_queue.cc:260] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdc63b471d240c3bc1eb9e7522ac089" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "1c61b194d8f041c8892564467ff18aa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44357 } } peers { permanent_uuid: "290922525e6a4b078aed0d7013736c15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40391 } }
I20260926 20:58:23.450009 13635 raft_consensus.cc:729] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1c61b194d8f041c8892564467ff18aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c61b194d8f041c8892564467ff18aa6, State: Initialized, Role: FOLLOWER
I20260926 20:58:23.450232 13638 ts_tablet_manager.cc:1428] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 20:58:23.450368 13467 heartbeater.cc:503] Master 127.0.98.62:45073 was elected leader, sending a full tablet report...
I20260926 20:58:23.450268 13635 consensus_queue.cc:260] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1c61b194d8f041c8892564467ff18aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdc63b471d240c3bc1eb9e7522ac089" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "1c61b194d8f041c8892564467ff18aa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44357 } } peers { permanent_uuid: "290922525e6a4b078aed0d7013736c15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40391 } }
I20260926 20:58:23.450676 13635 ts_tablet_manager.cc:1428] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1c61b194d8f041c8892564467ff18aa6: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 20:58:23.450836 13333 heartbeater.cc:503] Master 127.0.98.62:45073 was elected leader, sending a full tablet report...
I20260926 20:58:23.455180 13637 tablet_bootstrap.cc:492] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15: No bootstrap required, opened a new log
I20260926 20:58:23.455281 13637 ts_tablet_manager.cc:1397] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260926 20:58:23.457989 13637 raft_consensus.cc:348] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdc63b471d240c3bc1eb9e7522ac089" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "1c61b194d8f041c8892564467ff18aa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44357 } } peers { permanent_uuid: "290922525e6a4b078aed0d7013736c15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40391 } }
I20260926 20:58:23.458200 13637 raft_consensus.cc:374] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:23.458233 13637 raft_consensus.cc:729] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290922525e6a4b078aed0d7013736c15, State: Initialized, Role: FOLLOWER
I20260926 20:58:23.458391 13637 consensus_queue.cc:260] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdc63b471d240c3bc1eb9e7522ac089" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "1c61b194d8f041c8892564467ff18aa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44357 } } peers { permanent_uuid: "290922525e6a4b078aed0d7013736c15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40391 } }
I20260926 20:58:23.458684 13637 ts_tablet_manager.cc:1428] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 20:58:23.458815 13601 heartbeater.cc:503] Master 127.0.98.62:45073 was elected leader, sending a full tablet report...
W20260926 20:58:23.477908 13334 tablet.cc:2369] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1c61b194d8f041c8892564467ff18aa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:58:23.617743 13602 tablet.cc:2369] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 20:58:23 dist-test-slave-fl7g krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:58:23.665537 13641 raft_consensus.cc:482] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:58:23.665714 13641 raft_consensus.cc:504] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdc63b471d240c3bc1eb9e7522ac089" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "1c61b194d8f041c8892564467ff18aa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44357 } } peers { permanent_uuid: "290922525e6a4b078aed0d7013736c15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40391 } }
I20260926 20:58:23.666157 13641 leader_election.cc:288] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c61b194d8f041c8892564467ff18aa6 (127.0.98.1:44357), 290922525e6a4b078aed0d7013736c15 (127.0.98.3:40391)
I20260926 20:58:23.673388 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:23.666552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43915 (local address 127.0.98.1:44357)
0926 20:58:23.666591 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:23.666593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:23.667178 (+ 585us) server_negotiation.cc:407] Connection header received
0926 20:58:23.667303 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:23.667305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:23.667319 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:23.667349 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:23.667619 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.668341 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.672322 (+ 3981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.672946 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.672997 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:23.673010 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:23.673257 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 20:58:23.673309 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:58:23.673702 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:23.666299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44357 (local address 127.0.98.2:43915)
0926 20:58:23.667104 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:23.667117 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:23.667211 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:23.667425 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:23.667426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:23.667428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:23.667531 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:23.667534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.668480 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.668481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:23.669748 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:23.669752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.673073 (+ 3321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.673074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:23.673154 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:23.673157 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:58:23.673558 (+ 401us) client_negotiation.cc:240] Negotiation successful
0926 20:58:23.673607 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":107,"threads_started":1}
I20260926 20:58:23.674027 13287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29569e04b5e24d0ea5e5a2ace043bd2f" candidate_uuid: "1cdc63b471d240c3bc1eb9e7522ac089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c61b194d8f041c8892564467ff18aa6" is_pre_election: true
I20260926 20:58:23.674245 13287 raft_consensus.cc:2454] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1c61b194d8f041c8892564467ff18aa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cdc63b471d240c3bc1eb9e7522ac089 in term 0.
W20260926 20:58:23.674296 13468 tablet.cc:2369] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:58:23.674495 13354 leader_election.cc:302] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c61b194d8f041c8892564467ff18aa6, 1cdc63b471d240c3bc1eb9e7522ac089; no voters:
I20260926 20:58:23.674612 13641 raft_consensus.cc:2781] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:58:23.674666 13641 raft_consensus.cc:482] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:58:23.674698 13641 raft_consensus.cc:3037] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:23.675750 13641 raft_consensus.cc:504] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdc63b471d240c3bc1eb9e7522ac089" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "1c61b194d8f041c8892564467ff18aa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44357 } } peers { permanent_uuid: "290922525e6a4b078aed0d7013736c15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40391 } }
I20260926 20:58:23.675916 13641 leader_election.cc:288] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [CANDIDATE]: Term 1 election: Requested vote from peers 1c61b194d8f041c8892564467ff18aa6 (127.0.98.1:44357), 290922525e6a4b078aed0d7013736c15 (127.0.98.3:40391)
I20260926 20:58:23.676074 13643 raft_consensus.cc:482] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:58:23.676185 13643 raft_consensus.cc:504] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdc63b471d240c3bc1eb9e7522ac089" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "1c61b194d8f041c8892564467ff18aa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44357 } } peers { permanent_uuid: "290922525e6a4b078aed0d7013736c15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40391 } }
I20260926 20:58:23.676643 13643 leader_election.cc:288] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cdc63b471d240c3bc1eb9e7522ac089 (127.0.98.2:45135), 1c61b194d8f041c8892564467ff18aa6 (127.0.98.1:44357)
I20260926 20:58:23.677616 13287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29569e04b5e24d0ea5e5a2ace043bd2f" candidate_uuid: "1cdc63b471d240c3bc1eb9e7522ac089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c61b194d8f041c8892564467ff18aa6"
I20260926 20:58:23.677711 13287 raft_consensus.cc:3037] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1c61b194d8f041c8892564467ff18aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:23.678468 13287 raft_consensus.cc:2454] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1c61b194d8f041c8892564467ff18aa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cdc63b471d240c3bc1eb9e7522ac089 in term 1.
I20260926 20:58:23.678645 13354 leader_election.cc:302] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c61b194d8f041c8892564467ff18aa6, 1cdc63b471d240c3bc1eb9e7522ac089; no voters:
I20260926 20:58:23.678737 13641 raft_consensus.cc:2781] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:58:23.678879 13641 raft_consensus.cc:686] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [term 1 LEADER]: Becoming Leader. State: Replica: 1cdc63b471d240c3bc1eb9e7522ac089, State: Running, Role: LEADER
I20260926 20:58:23.678987 13641 consensus_queue.cc:237] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdc63b471d240c3bc1eb9e7522ac089" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "1c61b194d8f041c8892564467ff18aa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44357 } } peers { permanent_uuid: "290922525e6a4b078aed0d7013736c15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40391 } }
I20260926 20:58:23.679859 13135 catalog_manager.cc:5495] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cdc63b471d240c3bc1eb9e7522ac089 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1cdc63b471d240c3bc1eb9e7522ac089" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdc63b471d240c3bc1eb9e7522ac089" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c61b194d8f041c8892564467ff18aa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "290922525e6a4b078aed0d7013736c15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40391 } health_report { overall_health: UNKNOWN } } }
I20260926 20:58:23.681483 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:23.666519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60779 (local address 127.0.98.3:40391)
0926 20:58:23.666565 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:23.666567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:23.669946 (+ 3379us) server_negotiation.cc:407] Connection header received
0926 20:58:23.670060 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:23.670062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:23.670076 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:23.670105 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:23.670342 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.670989 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.680318 (+ 9329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.680901 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.680944 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:23.680952 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:23.681375 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 20:58:23.681421 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:58:23.681504 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:23.666350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40391 (local address 127.0.98.2:60779)
0926 20:58:23.669880 (+ 3530us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:23.669893 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:23.669980 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:23.670167 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:23.670168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:23.670170 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:23.670250 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:23.670252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.671084 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.671085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:23.672198 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:23.672201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.681144 (+ 8943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.681146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:23.681289 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:23.681293 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:58:23.681367 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:58:23.681414 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3507,"thread_start_us":60,"threads_started":1}
I20260926 20:58:23.681855 13555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29569e04b5e24d0ea5e5a2ace043bd2f" candidate_uuid: "1cdc63b471d240c3bc1eb9e7522ac089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290922525e6a4b078aed0d7013736c15" is_pre_election: true
I20260926 20:58:23.681988 13555 raft_consensus.cc:2454] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cdc63b471d240c3bc1eb9e7522ac089 in term 0.
I20260926 20:58:23.682092 13554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29569e04b5e24d0ea5e5a2ace043bd2f" candidate_uuid: "1cdc63b471d240c3bc1eb9e7522ac089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290922525e6a4b078aed0d7013736c15"
I20260926 20:58:23.682173 13554 raft_consensus.cc:3037] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:23.682888 13554 raft_consensus.cc:2454] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cdc63b471d240c3bc1eb9e7522ac089 in term 1.
I20260926 20:58:23.686995 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:23.677442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33977 (local address 127.0.98.1:44357)
0926 20:58:23.678676 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:23.678679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:23.683377 (+ 4698us) server_negotiation.cc:407] Connection header received
0926 20:58:23.683439 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:23.683440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:23.683457 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:23.683508 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:23.683704 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.684521 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.686048 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.686596 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.686643 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:23.686655 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:23.686889 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 20:58:23.686936 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1214}
I20260926 20:58:23.686990 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:23.677376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59063 (local address 127.0.98.2:45135)
0926 20:58:23.677439 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:23.677442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:23.683549 (+ 6107us) server_negotiation.cc:407] Connection header received
0926 20:58:23.683639 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:23.683640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:23.683652 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:23.683708 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:23.683931 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.684627 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.686047 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.686596 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.686648 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:23.686656 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:23.686875 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 20:58:23.686918 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 20:58:23.687279 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:23.677199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45135 (local address 127.0.98.3:59063)
0926 20:58:23.683494 (+ 6295us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:23.683503 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:23.683570 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:23.683797 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:23.683797 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:23.683798 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:23.683858 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:23.683859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.684725 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.684725 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:23.685937 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:23.685940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.686718 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.686719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:23.686789 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:23.686791 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:58:23.687144 (+ 353us) client_negotiation.cc:240] Negotiation successful
0926 20:58:23.687194 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6260,"mutex_wait_us":7,"thread_start_us":95,"threads_started":1}
I20260926 20:58:23.687584 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:23.676869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44357 (local address 127.0.98.3:33977)
0926 20:58:23.683297 (+ 6428us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:23.683307 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:23.683385 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:23.683553 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:23.683554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:23.683556 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:23.683625 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:23.683626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.684655 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.684656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:23.685937 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:23.685940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.686742 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.686743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:23.686807 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:23.686810 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:58:23.687299 (+ 489us) client_negotiation.cc:240] Negotiation successful
0926 20:58:23.687480 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6411,"mutex_wait_us":2,"thread_start_us":113,"threads_started":1}
I20260926 20:58:23.687578 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29569e04b5e24d0ea5e5a2ace043bd2f" candidate_uuid: "290922525e6a4b078aed0d7013736c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cdc63b471d240c3bc1eb9e7522ac089" is_pre_election: true
I20260926 20:58:23.688115 13287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29569e04b5e24d0ea5e5a2ace043bd2f" candidate_uuid: "290922525e6a4b078aed0d7013736c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c61b194d8f041c8892564467ff18aa6" is_pre_election: true
I20260926 20:58:23.688261 13287 raft_consensus.cc:2379] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1c61b194d8f041c8892564467ff18aa6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 290922525e6a4b078aed0d7013736c15 in current term 1: Already voted for candidate 1cdc63b471d240c3bc1eb9e7522ac089 in this term.
I20260926 20:58:23.688531 13488 leader_election.cc:302] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 290922525e6a4b078aed0d7013736c15; no voters: 1c61b194d8f041c8892564467ff18aa6, 1cdc63b471d240c3bc1eb9e7522ac089
I20260926 20:58:23.688766 13643 raft_consensus.cc:2726] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 20:58:23.712648 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:23.709280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.0.98.2:45135)
0926 20:58:23.709323 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:23.709325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:23.709333 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:58:23.709374 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:23.709375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:23.709396 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:23.709461 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:58:23.709662 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.710353 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.711420 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:23.711695 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:23.711775 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:23.711991 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:58:23.712132 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:58:23.712220 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:23.712539 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 20:58:23.712564 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:58:23.714986 13554 raft_consensus.cc:1261] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15 [term 1 FOLLOWER]: Refusing update from remote peer 1cdc63b471d240c3bc1eb9e7522ac089: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:58:23.715421 13641 consensus_queue.cc:1035] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [LEADER]: Connected to new peer: Peer: permanent_uuid: "290922525e6a4b078aed0d7013736c15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:58:23.717831 13287 raft_consensus.cc:1261] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1c61b194d8f041c8892564467ff18aa6 [term 1 FOLLOWER]: Refusing update from remote peer 1cdc63b471d240c3bc1eb9e7522ac089: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:58:23.718159 13641 consensus_queue.cc:1035] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61b194d8f041c8892564467ff18aa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:58:23.732504 13653 mvcc.cc:205] Tried to move back new op lower bound from 7333709020013359104 to 7333709019870003200. Current Snapshot: MvccSnapshot[applied={T|T < 7333709020013359104}]
I20260926 20:58:23.734148 13656 mvcc.cc:205] Tried to move back new op lower bound from 7333709020013359104 to 7333709019870003200. Current Snapshot: MvccSnapshot[applied={T|T < 7333709020013359104}]
I20260926 20:58:23.739208 13654 mvcc.cc:205] Tried to move back new op lower bound from 7333709020013359104 to 7333709019870003200. Current Snapshot: MvccSnapshot[applied={T|T < 7333709020013359104}]
I20260926 20:58:23.742074 13135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:58:23.742205 13135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:58:23.742967 13135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1da80bbb86c04ff788d339841a01bb3e: Sending DeleteTablet for 3 replicas of tablet 29569e04b5e24d0ea5e5a2ace043bd2f
I20260926 20:58:23.743463 13267 tablet_service.cc:1514] Processing DeleteTablet for tablet 29569e04b5e24d0ea5e5a2ace043bd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:58:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59890
I20260926 20:58:23.743746 13535 tablet_service.cc:1514] Processing DeleteTablet for tablet 29569e04b5e24d0ea5e5a2ace043bd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:58:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57474
I20260926 20:58:23.744128 13401 tablet_service.cc:1514] Processing DeleteTablet for tablet 29569e04b5e24d0ea5e5a2ace043bd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:58:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46182
I20260926 20:58:23.744766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 13174
I20260926 20:58:23.756057 13665 tablet_replica.cc:330] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089: stopping tablet replica
I20260926 20:58:23.756249 13665 raft_consensus.cc:2229] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:58:23.756458 13665 raft_consensus.cc:2258] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:58:23.756857 13664 tablet_replica.cc:330] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15: stopping tablet replica
I20260926 20:58:23.756992 13664 raft_consensus.cc:2229] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:58:23.757110 13664 raft_consensus.cc:2258] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:58:23.757627 13665 ts_tablet_manager.cc:1905] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:58:23.759608 13119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44357 (error 108)
I20260926 20:58:23.759980 13665 ts_tablet_manager.cc:1918] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:58:23.760063 13665 log.cc:1196] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/29569e04b5e24d0ea5e5a2ace043bd2f
I20260926 20:58:23.760296 13664 ts_tablet_manager.cc:1905] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:58:23.760332 13665 ts_tablet_manager.cc:1939] T 29569e04b5e24d0ea5e5a2ace043bd2f P 1cdc63b471d240c3bc1eb9e7522ac089: Deleting consensus metadata
I20260926 20:58:23.761272 13122 catalog_manager.cc:4841] TS 1cdc63b471d240c3bc1eb9e7522ac089 (127.0.98.2:45135): tablet 29569e04b5e24d0ea5e5a2ace043bd2f (table test-table [id=67c455d91e1242ecb793ce5bb1439403]) successfully deleted
I20260926 20:58:23.761490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 13337
I20260926 20:58:23.762651 13664 ts_tablet_manager.cc:1918] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:58:23.762852 13664 log.cc:1196] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/29569e04b5e24d0ea5e5a2ace043bd2f
I20260926 20:58:23.763176 13664 ts_tablet_manager.cc:1939] T 29569e04b5e24d0ea5e5a2ace043bd2f P 290922525e6a4b078aed0d7013736c15: Deleting consensus metadata
I20260926 20:58:23.763676 13121 catalog_manager.cc:4841] TS 290922525e6a4b078aed0d7013736c15 (127.0.98.3:40391): tablet 29569e04b5e24d0ea5e5a2ace043bd2f (table test-table [id=67c455d91e1242ecb793ce5bb1439403]) successfully deleted
I20260926 20:58:23.769011 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:23.768837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44357 (local address 127.0.0.1:59902)
0926 20:58:23.768911 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:23.768959 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 20:58:23.769217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 13470
W20260926 20:58:23.770542 13119 catalog_manager.cc:4584] TS 1c61b194d8f041c8892564467ff18aa6 (127.0.98.1:44357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29569e04b5e24d0ea5e5a2ace043bd2f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44357: connect: Connection refused (error 111)
I20260926 20:58:23.783272 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 13102
I20260926 20:58:23.791050 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 20:58:24.136332 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 20:58:24.136523 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 20:58:24.481863 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 20:58:24.482 UTC [12169] LOG: received smart shutdown request
2026-09-26 20:58:24.486 UTC [12169] LOG: background worker "logical replication launcher" (PID 12178) exited with exit code 1
2026-09-26 20:58:24.486 UTC [12173] LOG: shutting down
2026-09-26 20:58:24.700 UTC [12169] LOG: database system is shut down
2026-09-26T20:58:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49763 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:58:24 dist-test-slave-fl7g krb5kdc[13675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:58:24 dist-test-slave-fl7g krb5kdc[13675](info): set up 2 sockets
Sep 26 20:58:24 dist-test-slave-fl7g krb5kdc[13675](info): commencing operation
krb5kdc: starting...
W20260926 20:58:25.868673 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:58:25 dist-test-slave-fl7g krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:58:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:58:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:58:25.995471 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34361
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34361,127.0.98.61:40587,127.0.98.60:40261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:58:26.079272 13688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:26.079552 13688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:26.079593 13688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:26.080837 13688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:58:26.080921 13688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:26.080945 13688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:26.080967 13688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:58:26.080999 13688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:58:26.082684 13688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34361,127.0.98.61:40587,127.0.98.60:40261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:26.083004 13688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:26.083261 13688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:26.088886 13696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:26.088848 13693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:26.088985 13688 server_base.cc:1034] running on GCE node
W20260926 20:58:26.088848 13694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:26.089597 13688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:26.089895 13688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:26.091058 13688 hybrid_clock.cc:648] HybridClock initialized: now 1790456306091042 us; error 35 us; skew 500 ppm
Sep 26 20:58:26 dist-test-slave-fl7g krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:26.093626 13688 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:58:26.094159 13688 webserver.cc:460] Webserver started at http://127.0.98.62:42391/ using document root <none> and password file <none>
I20260926 20:58:26.094401 13688 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:26.094470 13688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:26.094628 13688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:58:26.095732 13688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6"
format_stamp: "Formatted at 2026-09-26 20:58:26 on dist-test-slave-fl7g"
server_key: "62508b41c2700ab07caa7d77c81d3d24"
server_key_iv: "8b69cf066a72fd215e5cb92686f8f9a2"
server_key_version: "encryptionkey@0"
I20260926 20:58:26.096113 13688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6"
format_stamp: "Formatted at 2026-09-26 20:58:26 on dist-test-slave-fl7g"
server_key: "62508b41c2700ab07caa7d77c81d3d24"
server_key_iv: "8b69cf066a72fd215e5cb92686f8f9a2"
server_key_version: "encryptionkey@0"
I20260926 20:58:26.097587 13688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:58:26.098434 13703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:26.098694 13688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:58:26.098820 13688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6"
format_stamp: "Formatted at 2026-09-26 20:58:26 on dist-test-slave-fl7g"
server_key: "62508b41c2700ab07caa7d77c81d3d24"
server_key_iv: "8b69cf066a72fd215e5cb92686f8f9a2"
server_key_version: "encryptionkey@0"
I20260926 20:58:26.098902 13688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:26.124127 13688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:26.126380 13688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:26.126560 13688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:26.130860 13688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34361
I20260926 20:58:26.130897 13755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34361 every 8 connection(s)
I20260926 20:58:26.131301 13688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:58:26.132207 13756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:26.133874 13756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:58:26 dist-test-slave-fl7g krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:26.141306 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 13688
I20260926 20:58:26.141419 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:58:26.141607 392 external_mini_cluster.cc:1428] Setting key 487aa16be85a209a5680575de237170e
I20260926 20:58:26.143947 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.134605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59126 (local address 127.0.98.62:34361)
0926 20:58:26.134808 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:26.134812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:26.134885 (+ 73us) server_negotiation.cc:407] Connection header received
0926 20:58:26.135404 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:26.135406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:26.135422 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:26.135498 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:26.136040 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.136912 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.137770 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.138084 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.138179 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.140309 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:26.140316 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:26.140316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:26.140322 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:26.142197 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.142664 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.142667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.142667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.142753 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.143020 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.143021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.143022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.143254 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:26.143354 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:26.143535 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 20:58:26.143686 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":114,"threads_started":1}
I20260926 20:58:26.143947 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.134564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34361 (local address 127.0.0.1:59126)
0926 20:58:26.134735 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.134792 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:26.135278 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:26.135524 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:26.135527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:26.135795 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:26.135954 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:26.135958 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.136997 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.136997 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:26.137688 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:26.137689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.138161 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.138162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:26.138229 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.138761 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:26.138769 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:26.140224 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:26.142279 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:26.142286 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:26.142289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:26.142562 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:26.142852 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:26.142853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:26.142854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:26.142946 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:26.143329 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:26.143334 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:26.143403 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 20:58:26.143502 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 20:58:26.143749 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":96,"thread_start_us":70,"threads_started":1}
I20260926 20:58:26.144767 13756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:58:26.145123 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.145005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40587 (local address 127.0.0.1:36646)
0926 20:58:26.145056 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.145100 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 20:58:26.145251 13706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40587: connect: Connection refused (error 111)
I20260926 20:58:26.145712 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.145602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40587 (local address 127.0.0.1:36660)
0926 20:58:26.145672 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.145694 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 20:58:26.145807 13756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40587: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40587: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 20:58:26.168144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40587
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34361,127.0.98.61:40587,127.0.98.60:40261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:58:26.195040 13756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } attempt: 1
I20260926 20:58:26.195591 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.195436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40587 (local address 127.0.0.1:36670)
0926 20:58:26.195508 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.195552 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 20:58:26.195963 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.195863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40587 (local address 127.0.0.1:36680)
0926 20:58:26.195912 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.195937 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 20:58:26.196053 13756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40587: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40587: connect: Connection refused (error 111)
W20260926 20:58:26.252694 13763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:26.252997 13763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:26.253043 13763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:26.254299 13763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:58:26.254374 13763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:26.254398 13763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:26.254420 13763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:58:26.254451 13763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:58:26.256075 13763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34361,127.0.98.61:40587,127.0.98.60:40261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:26.256438 13763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:26.256721 13763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:26.261952 13768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:26.262022 13769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:26.261960 13771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:26.262167 13763 server_base.cc:1034] running on GCE node
I20260926 20:58:26.262504 13763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:26.262831 13763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:26.264030 13763 hybrid_clock.cc:648] HybridClock initialized: now 1790456306264022 us; error 42 us; skew 500 ppm
I20260926 20:58:26.264232 13756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } attempt: 2
I20260926 20:58:26.264843 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.264622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40587 (local address 127.0.0.1:36692)
0926 20:58:26.264776 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.264812 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260926 20:58:26.265257 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.265151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40587 (local address 127.0.0.1:36702)
0926 20:58:26.265200 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.265230 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 20:58:26.265364 13756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40587: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40587: connect: Connection refused (error 111)
Sep 26 20:58:26 dist-test-slave-fl7g krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:26.267024 13763 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 20:58:26.267549 13763 webserver.cc:460] Webserver started at http://127.0.98.61:44997/ using document root <none> and password file <none>
I20260926 20:58:26.267807 13763 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:26.267874 13763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:26.268024 13763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:58:26.269353 13763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-1/data/instance:
uuid: "66b1993369b34e3888577a3ce1ed30e3"
format_stamp: "Formatted at 2026-09-26 20:58:26 on dist-test-slave-fl7g"
server_key: "ea0eaacef0038bf055f13a7e775f1b81"
server_key_iv: "3b0869f9aa7c09c62558aedf56904daf"
server_key_version: "encryptionkey@0"
I20260926 20:58:26.269799 13763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-1/wal/instance:
uuid: "66b1993369b34e3888577a3ce1ed30e3"
format_stamp: "Formatted at 2026-09-26 20:58:26 on dist-test-slave-fl7g"
server_key: "ea0eaacef0038bf055f13a7e775f1b81"
server_key_iv: "3b0869f9aa7c09c62558aedf56904daf"
server_key_version: "encryptionkey@0"
I20260926 20:58:26.271275 13763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:58:26.272130 13778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:26.272363 13763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:26.272496 13763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-1/wal
uuid: "66b1993369b34e3888577a3ce1ed30e3"
format_stamp: "Formatted at 2026-09-26 20:58:26 on dist-test-slave-fl7g"
server_key: "ea0eaacef0038bf055f13a7e775f1b81"
server_key_iv: "3b0869f9aa7c09c62558aedf56904daf"
server_key_version: "encryptionkey@0"
I20260926 20:58:26.272578 13763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:26.307158 13763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:26.309507 13763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:26.309690 13763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:26.313905 13763 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40587
I20260926 20:58:26.313941 13830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40587 every 8 connection(s)
I20260926 20:58:26.314301 13763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-1/data/info.pb
I20260926 20:58:26.315157 13831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:26.316805 13831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:58:26 dist-test-slave-fl7g krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:26.324190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 13763
I20260926 20:58:26.324319 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-1/wal/instance
I20260926 20:58:26.324572 392 external_mini_cluster.cc:1428] Setting key c02480e4da29a1da7fdb10545d7531ab
I20260926 20:58:26.326484 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.317583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59136 (local address 127.0.98.62:34361)
0926 20:58:26.317662 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:26.317663 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:26.317876 (+ 213us) server_negotiation.cc:407] Connection header received
0926 20:58:26.318357 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:26.318358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:26.318366 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:26.318432 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:26.319042 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.319537 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.320513 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.320712 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.320770 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.323090 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:26.323091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:26.323091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:26.323098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:26.324841 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.325285 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.325287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.325287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.325362 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.325609 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.325610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.325611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.325846 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:26.325927 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:26.326290 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 20:58:26.326390 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:58:26.326697 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.317549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34361 (local address 127.0.0.1:59136)
0926 20:58:26.317740 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.317784 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:26.318272 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:26.318499 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:26.318502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:26.318763 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:26.318966 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:26.318970 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.319617 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.319618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:26.320436 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:26.320438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.320819 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.320820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:26.320948 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.321543 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:26.321552 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:26.322989 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:26.324943 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:26.324950 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:26.324952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:26.325189 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:26.325433 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:26.325433 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:26.325434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:26.325539 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:26.325971 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:26.325977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:26.326119 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 20:58:26.326266 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 20:58:26.326444 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":110,"threads_started":1}
I20260926 20:58:26.327335 13831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:58:26 dist-test-slave-fl7g krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260926 20:58:26.335654 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.327651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40587 (local address 127.0.0.1:36718)
0926 20:58:26.327714 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.327727 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:26.327775 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:26.328117 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:26.328119 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:26.328282 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:26.328333 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:26.328335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.329422 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.329424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:26.329991 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:26.329993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.330545 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.330547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:26.330631 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.331142 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:26.331147 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:26.332512 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:26.334408 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:26.334409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:26.334409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:26.334597 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:26.334850 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:26.334851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:26.334852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:26.334919 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:26.335243 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:26.335244 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:26.335286 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 20:58:26.335357 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:58:26.335472 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 20:58:26.335693 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.327761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.0.98.61:40587)
0926 20:58:26.327939 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:26.327943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:26.327961 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:58:26.327980 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:26.327981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:26.327992 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:26.328115 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:26.328440 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.329331 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.330110 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.330448 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.330548 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.332622 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:26.332639 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:26.332640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:26.332666 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:26.334293 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.334702 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.334707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.334708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.334765 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.334997 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.334997 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.334998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.335162 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:26.335268 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:26.335402 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 20:58:26.335553 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":153,"thread_start_us":94,"threads_started":1}
I20260926 20:58:26.336335 13831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:58:26.336790 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.336652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40261 (local address 127.0.0.1:37268)
0926 20:58:26.336707 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.336759 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 20:58:26.337001 13781 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
I20260926 20:58:26.337472 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.337369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40261 (local address 127.0.0.1:37274)
0926 20:58:26.337420 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.337447 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 20:58:26.337589 13831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40261: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 20:58:26.351841 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40261
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34361,127.0.98.61:40587,127.0.98.60:40261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:58:26.356489 13756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } attempt: 3
Sep 26 20:58:26 dist-test-slave-fl7g krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 20:58:26.364372 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.356975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36726 (local address 127.0.98.61:40587)
0926 20:58:26.357045 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:26.357047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:26.357058 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:58:26.357115 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:26.357116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:26.357131 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:26.357196 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:26.357527 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.358039 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.358743 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.358954 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.359043 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.361490 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:26.361492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:26.361492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:26.361499 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:26.362992 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.363467 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.363468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.363468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.363524 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.363811 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.363812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.363812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.363982 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:26.364048 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:26.364228 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 20:58:26.364302 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:58:26.364543 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.356864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40587 (local address 127.0.0.1:36726)
0926 20:58:26.356951 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.356972 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:26.357043 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:26.357217 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:26.357218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:26.357374 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:26.357437 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:26.357438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.358151 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.358152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:26.358658 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:26.358659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.359043 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.359044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:26.359130 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.359611 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:26.359616 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:26.361377 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:26.363097 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:26.363098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:26.363099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:26.363369 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:26.363602 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:26.363603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:26.363604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:26.363688 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:26.364046 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:26.364048 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:26.364110 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 20:58:26.364203 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 20:58:26.364323 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 20:58:26.364995 13756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:58:26.365411 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.365277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40261 (local address 127.0.0.1:37276)
0926 20:58:26.365335 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.365378 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 20:58:26.365839 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.365737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40261 (local address 127.0.0.1:37290)
0926 20:58:26.365784 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.365816 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 20:58:26.365994 13756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40261: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
I20260926 20:58:26.386876 13831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } attempt: 1
I20260926 20:58:26.387473 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.387307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40261 (local address 127.0.0.1:37292)
0926 20:58:26.387381 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.387434 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 20:58:26.387916 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.387790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40261 (local address 127.0.0.1:37296)
0926 20:58:26.387848 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.387892 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 20:58:26.388072 13831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40261: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
I20260926 20:58:26.397164 13756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } attempt: 1
I20260926 20:58:26.397706 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.397549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40261 (local address 127.0.0.1:37306)
0926 20:58:26.397617 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.397669 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 20:58:26.398099 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.397991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40261 (local address 127.0.0.1:37310)
0926 20:58:26.398047 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.398079 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 20:58:26.398249 13756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40261: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
W20260926 20:58:26.443925 13838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:26.444187 13838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:26.444216 13838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:26.445559 13838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:58:26.445622 13838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:26.445640 13838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:26.445655 13838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:58:26.445669 13838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:58:26.447382 13838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34361,127.0.98.61:40587,127.0.98.60:40261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:26.447701 13838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:26.447968 13838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:26.453325 13844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:26.453356 13843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:26.453564 13846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:26.453603 13838 server_base.cc:1034] running on GCE node
I20260926 20:58:26.453894 13838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:26.454213 13838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:26.455387 13838 hybrid_clock.cc:648] HybridClock initialized: now 1790456306455381 us; error 63 us; skew 500 ppm
I20260926 20:58:26.456260 13831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } attempt: 2
I20260926 20:58:26.456739 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.456600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40261 (local address 127.0.0.1:37320)
0926 20:58:26.456666 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.456697 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 20:58:26.457141 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.457044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40261 (local address 127.0.0.1:37336)
0926 20:58:26.457079 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.457112 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260926 20:58:26.457234 13831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40261: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
Sep 26 20:58:26 dist-test-slave-fl7g krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:26.458321 13838 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 20:58:26.458856 13838 webserver.cc:460] Webserver started at http://127.0.98.60:41767/ using document root <none> and password file <none>
I20260926 20:58:26.459105 13838 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:26.459151 13838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:26.459290 13838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:58:26.460475 13838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-2/data/instance:
uuid: "1196fa070c76435999c32d176a2d8733"
format_stamp: "Formatted at 2026-09-26 20:58:26 on dist-test-slave-fl7g"
server_key: "27d9be49098a58c2e9c010f7810d2d57"
server_key_iv: "524e9ced9c6e9283200a59b63e2182de"
server_key_version: "encryptionkey@0"
I20260926 20:58:26.460865 13838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-2/wal/instance:
uuid: "1196fa070c76435999c32d176a2d8733"
format_stamp: "Formatted at 2026-09-26 20:58:26 on dist-test-slave-fl7g"
server_key: "27d9be49098a58c2e9c010f7810d2d57"
server_key_iv: "524e9ced9c6e9283200a59b63e2182de"
server_key_version: "encryptionkey@0"
I20260926 20:58:26.462297 13838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:58:26.463201 13853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:26.463408 13838 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:26.463503 13838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-2/wal
uuid: "1196fa070c76435999c32d176a2d8733"
format_stamp: "Formatted at 2026-09-26 20:58:26 on dist-test-slave-fl7g"
server_key: "27d9be49098a58c2e9c010f7810d2d57"
server_key_iv: "524e9ced9c6e9283200a59b63e2182de"
server_key_version: "encryptionkey@0"
I20260926 20:58:26.463560 13838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:26.472144 13838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:26.473413 13756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } attempt: 2
I20260926 20:58:26.473881 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.473753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40261 (local address 127.0.0.1:37350)
0926 20:58:26.473814 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.473851 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 20:58:26.474303 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.474186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40261 (local address 127.0.0.1:37360)
0926 20:58:26.474248 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.474284 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 20:58:26.474428 13756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40261: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40261: connect: Connection refused (error 111)
I20260926 20:58:26.474733 13838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:26.474897 13838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:26.479228 13838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40261
I20260926 20:58:26.479276 13905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40261 every 8 connection(s)
I20260926 20:58:26.479677 13838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-2/data/info.pb
I20260926 20:58:26.480701 13906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:26.482438 13906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:58:26.487315 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 13838
I20260926 20:58:26.487406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790456150796563-392-0/minicluster-data/master-2/wal/instance
I20260926 20:58:26.487643 392 external_mini_cluster.cc:1428] Setting key 0df3946323a072e8c3ea3addab27077d
Sep 26 20:58:26 dist-test-slave-fl7g krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:26.492834 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.483268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59150 (local address 127.0.98.62:34361)
0926 20:58:26.483349 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:26.483351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:26.483580 (+ 229us) server_negotiation.cc:407] Connection header received
0926 20:58:26.484101 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:26.484102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:26.484121 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:26.484200 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:26.484969 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.485479 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.486552 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.486734 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.486788 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.489540 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:26.489542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:26.489542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:26.489578 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:26.491209 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.491680 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.491681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.491682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.491749 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.492033 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.492034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.492034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.492169 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:26.492224 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:26.492624 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 20:58:26.492709 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:58:26.492877 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.483253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34361 (local address 127.0.0.1:59150)
0926 20:58:26.483435 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.483486 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:26.483972 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:26.484285 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:26.484289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:26.484641 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:26.484857 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:26.484861 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.485586 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.485587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:26.486473 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:26.486475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.486828 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.486829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:26.486947 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.487677 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:26.487690 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:26.489443 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:26.491301 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:26.491309 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:26.491311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:26.491603 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:26.491819 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:26.491820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:26.491821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:26.491964 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:26.492242 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:26.492249 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:26.492463 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 20:58:26.492571 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 20:58:26.492704 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260926 20:58:26.493526 13906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:58:26 dist-test-slave-fl7g krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 20:58:26.497732 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.488091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37368 (local address 127.0.98.60:40261)
0926 20:58:26.488274 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:26.488278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:26.488294 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:58:26.488352 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:26.488354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:26.488371 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:26.488513 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:26.489168 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.490167 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.491301 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.491763 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.491851 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.494698 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:26.494724 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:26.494724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:26.494732 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:26.496544 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.496921 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.496924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.496925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.496978 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.497175 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.497176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.497176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.497365 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:26.497473 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:26.497593 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 20:58:26.497653 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
Sep 26 20:58:26 dist-test-slave-fl7g krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 20:58:26.501471 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.493810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40587 (local address 127.0.0.1:36740)
0926 20:58:26.493949 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.493960 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:26.494018 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:26.494152 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:26.494153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:26.494308 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:26.494358 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:26.494360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.495123 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.495124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:26.495817 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:26.495820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.496271 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.496272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:26.496329 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.496753 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:26.496757 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:26.498202 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:26.500075 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:26.500076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:26.500077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:26.500262 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:26.500520 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:26.500522 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:26.500522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:26.500587 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:26.501118 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:26.501119 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:26.501170 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 20:58:26.501262 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:58:26.501367 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260926 20:58:26.501506 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.493891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.0.98.61:40587)
0926 20:58:26.493968 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:26.493970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:26.494003 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:58:26.494087 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:26.494088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:26.494097 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:26.494138 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:26.494436 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.495050 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.495899 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.496182 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.496276 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.498357 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:26.498358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:26.498358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:26.498366 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:26.499963 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.500341 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.500342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.500342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.500457 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.500816 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.500817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.500818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.501019 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:26.501119 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:26.501290 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 20:58:26.501368 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:58:26.502090 13906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:58:26 dist-test-slave-fl7g krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 20:58:26 dist-test-slave-fl7g krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 20:58:26.506587 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.498786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59160 (local address 127.0.98.62:34361)
0926 20:58:26.498838 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:26.498840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:26.498850 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:58:26.498933 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:26.498934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:26.498942 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:26.499015 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:26.499515 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.500003 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.500771 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.500978 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.501066 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.503565 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:26.503567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:26.503567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:26.503575 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:26.505349 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.505726 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.505728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.505728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.505789 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.505994 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.505994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.505995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.506128 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:26.506354 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:26.506393 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 20:58:26.506468 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 20:58:26.509727 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.502354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40261 (local address 127.0.0.1:37384)
0926 20:58:26.502403 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.502413 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:26.502452 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:26.502584 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:26.502584 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:26.502723 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:26.502754 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:26.502755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.503502 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.503503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:26.504284 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:26.504287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.504708 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.504709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:26.504757 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.505114 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:26.505119 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:26.506398 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:26.508666 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:26.508667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:26.508668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:26.508881 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:26.509077 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:26.509077 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:26.509078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:26.509142 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:26.509488 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:26.509488 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:26.509532 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 20:58:26.509590 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:58:26.509660 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 20:58:26.509852 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.502420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37384 (local address 127.0.98.60:40261)
0926 20:58:26.502454 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:26.502456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:26.502463 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:58:26.502502 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:26.502503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:26.502513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:26.502559 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:26.502824 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.503427 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.504349 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.504624 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.504709 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.506511 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:26.506512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:26.506513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:26.506520 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:26.508568 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.508958 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.508958 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.508959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.509012 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.509268 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.509269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.509269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.509383 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:26.509496 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:26.509643 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 20:58:26.509755 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:58:26.511520 13906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733: Bootstrap starting.
I20260926 20:58:26.512482 13906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733: Neither blocks nor log segments found. Creating new log.
Sep 26 20:58:26 dist-test-slave-fl7g krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 20:58:26.512838 13906 log.cc:824] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:26.513684 13906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733: No bootstrap required, opened a new log
I20260926 20:58:26.515286 13906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } }
I20260926 20:58:26.515445 13906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:26.515496 13906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1196fa070c76435999c32d176a2d8733, State: Initialized, Role: FOLLOWER
I20260926 20:58:26.515645 13906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } }
I20260926 20:58:26.515755 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.507791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.0.98.61:40587)
0926 20:58:26.507849 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:26.507851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:26.507860 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:58:26.507882 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:26.507883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:26.507893 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:26.507944 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:26.508666 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.509393 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.510434 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.510658 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.510749 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.513194 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:26.513195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:26.513196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:26.513212 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:26.514594 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.514947 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.514949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.514949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.515009 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.515218 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.515219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.515219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.515338 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:26.515539 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:26.515576 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 20:58:26.515655 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:58:26.515995 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } } }
I20260926 20:58:26.516062 13906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:58:26.516078 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:58:26.517663 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 20:58:26.518072 13923 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:58:26.518121 13923 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:58:26.519629 13838 master_runner.cc:186] Error getting master registration for 127.0.98.62:34361: OK, instance_id { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" instance_seqno: 1790456306129627 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:58:26.522660 13838 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:58:26.527645 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.519462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.0.98.62:34361)
0926 20:58:26.519558 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:26.519560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:26.519568 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:58:26.519590 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:26.519591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:26.519602 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:26.519653 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:26.520318 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.521159 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.522179 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.522462 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.522523 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.524256 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:26.524258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:26.524258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:26.524265 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:26.526121 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.526508 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.526510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.526510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.526568 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.526785 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.526786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.526787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.526929 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:26.527046 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:26.527456 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 20:58:26.527538 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 20:58:26.528277 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.519375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36770 (local address 127.0.98.61:40587)
0926 20:58:26.519414 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:26.519416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:26.520404 (+ 988us) server_negotiation.cc:407] Connection header received
0926 20:58:26.520548 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:26.520549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:26.520559 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:26.520588 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:26.521270 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.521976 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.522919 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.523230 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.523276 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.524868 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:26.524869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:26.524869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:26.524877 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:26.526752 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.527139 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.527140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.527140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.527239 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.527453 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.527454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.527455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.527582 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:26.527648 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:26.528125 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 20:58:26.528207 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:58:26.530177 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.519652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.0.98.60:40261)
0926 20:58:26.519698 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:26.519700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:26.522848 (+ 3148us) server_negotiation.cc:407] Connection header received
0926 20:58:26.522947 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:26.522948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:26.522958 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:26.522989 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:26.523748 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.524513 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.525504 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.525786 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.525831 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.527294 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:26.527295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:26.527296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:26.527303 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:26.529012 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.529426 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.529427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.529427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.529483 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.529691 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.529691 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.529691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.529824 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:26.529913 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:26.530053 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 20:58:26.530117 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:58:26.548391 13831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } attempt: 3
Sep 26 20:58:26 dist-test-slave-fl7g krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 20:58:26.555472 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.548867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.0.98.60:40261)
0926 20:58:26.548932 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:26.548934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:26.548945 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:58:26.549017 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:26.549018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:26.549029 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:26.549098 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:26.549414 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.549882 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.550548 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.550740 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.550825 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.553001 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:26.553003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:26.553003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:26.553011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:26.554387 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.554797 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.554798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.554798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.554848 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.555036 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.555037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.555037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.555140 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:26.555189 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:26.555346 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 20:58:26.555404 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:58:26.555537 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.548798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40261 (local address 127.0.0.1:37396)
0926 20:58:26.548865 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.548882 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:26.548952 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:26.549117 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:26.549118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:26.549292 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:26.549335 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:26.549337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.549973 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.549975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:26.550467 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:26.550468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.550826 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.550827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:26.550892 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.551392 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:26.551397 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:26.552924 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:26.554474 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:26.554475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:26.554475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:26.554717 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:26.554908 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:26.554909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:26.554909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:26.554976 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:26.555200 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:26.555201 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:26.555253 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 20:58:26.555331 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:58:26.555433 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 20:58:26.557037 13831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3: Bootstrap starting.
I20260926 20:58:26.557751 13831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3: Neither blocks nor log segments found. Creating new log.
I20260926 20:58:26.558058 13831 log.cc:824] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:26.558846 13831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3: No bootstrap required, opened a new log
I20260926 20:58:26.560330 13831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } }
I20260926 20:58:26.560534 13831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:26.560586 13831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b1993369b34e3888577a3ce1ed30e3, State: Initialized, Role: FOLLOWER
I20260926 20:58:26.560727 13831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } }
I20260926 20:58:26.561038 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } } }
I20260926 20:58:26.561115 13831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:58:26.561141 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:58:26.563117 13944 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:58:26.563197 13944 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:58:26.563277 13763 master_runner.cc:186] Error getting master registration for 127.0.98.62:34361: OK, instance_id { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" instance_seqno: 1790456306129627 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:58:26.565873 13763 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:58:26.573585 13756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } attempt: 3
Sep 26 20:58:26 dist-test-slave-fl7g krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 20:58:26.582188 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.573949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40261 (local address 127.0.0.1:37404)
0926 20:58:26.573997 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:26.574009 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:26.574062 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:26.574233 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:26.574234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:26.574437 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:26.574497 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:26.574499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.575486 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.575487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:26.576365 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:26.576368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.576874 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.576875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:26.576940 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.577481 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:26.577486 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:26.579077 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:26.581026 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:26.581027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:26.581027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:26.581263 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:26.581483 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:26.581484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:26.581484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:26.581552 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:26.581872 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:26.581874 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:26.581924 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 20:58:26.581986 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 20:58:26.582089 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 20:58:26.582727 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:26.574006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.0.98.60:40261)
0926 20:58:26.574096 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:26.574099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:26.574108 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:58:26.574133 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:26.574134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:26.574144 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:26.574201 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:26.574590 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.575396 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.576473 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:26.576792 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:26.576866 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:26.579171 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:26.579173 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:26.579173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:26.579181 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:26.580936 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.581342 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.581343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.581343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.581409 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:26.581625 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:26.581626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:26.581626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:26.581772 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:26.581840 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:26.582541 (+ 701us) server_negotiation.cc:299] Negotiation successful
0926 20:58:26.582636 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:58:26.584647 13756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6: Bootstrap starting.
I20260926 20:58:26.585427 13756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6: Neither blocks nor log segments found. Creating new log.
I20260926 20:58:26.585745 13756 log.cc:824] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:26.586678 13756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6: No bootstrap required, opened a new log
I20260926 20:58:26.588742 13756 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } }
I20260926 20:58:26.588898 13756 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:26.588948 13756 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7759eaf3a5c40da83ca0902f2d7d6f6, State: Initialized, Role: FOLLOWER
I20260926 20:58:26.589077 13756 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } }
I20260926 20:58:26.589421 13756 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:58:26.589383 13947 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } } }
I20260926 20:58:26.589592 13947 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:58:26.591324 13958 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:58:26.591446 13958 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:58:26.593868 13688 master_runner.cc:263] No leader master found from master b7759eaf3a5c40da83ca0902f2d7d6f6
I20260926 20:58:26.593921 13688 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 20:58:26.627490 13709 tablet.cc:2369] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:58:26.725682 13859 tablet.cc:2369] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:58:26.810643 13784 tablet.cc:2369] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:58:26.866931 13947 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:58:26.867060 13947 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } }
I20260926 20:58:26.867324 13947 leader_election.cc:288] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66b1993369b34e3888577a3ce1ed30e3 (127.0.98.61:40587), 1196fa070c76435999c32d176a2d8733 (127.0.98.60:40261)
I20260926 20:58:26.867683 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66b1993369b34e3888577a3ce1ed30e3" is_pre_election: true
I20260926 20:58:26.867731 13880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1196fa070c76435999c32d176a2d8733" is_pre_election: true
I20260926 20:58:26.867858 13805 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7759eaf3a5c40da83ca0902f2d7d6f6 in term 0.
I20260926 20:58:26.867910 13880 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7759eaf3a5c40da83ca0902f2d7d6f6 in term 0.
I20260926 20:58:26.868112 13706 leader_election.cc:302] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1196fa070c76435999c32d176a2d8733, b7759eaf3a5c40da83ca0902f2d7d6f6; no voters:
I20260926 20:58:26.868263 13947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:58:26.868325 13947 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:58:26.868361 13947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:26.869225 13947 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } }
I20260926 20:58:26.869400 13947 leader_election.cc:288] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [CANDIDATE]: Term 1 election: Requested vote from peers 66b1993369b34e3888577a3ce1ed30e3 (127.0.98.61:40587), 1196fa070c76435999c32d176a2d8733 (127.0.98.60:40261)
I20260926 20:58:26.869608 13880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1196fa070c76435999c32d176a2d8733"
I20260926 20:58:26.869699 13880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:26.870357 13880 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7759eaf3a5c40da83ca0902f2d7d6f6 in term 1.
I20260926 20:58:26.870497 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66b1993369b34e3888577a3ce1ed30e3"
I20260926 20:58:26.870559 13706 leader_election.cc:302] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1196fa070c76435999c32d176a2d8733, b7759eaf3a5c40da83ca0902f2d7d6f6; no voters:
I20260926 20:58:26.870622 13805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:26.870713 13947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:58:26.870849 13947 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [term 1 LEADER]: Becoming Leader. State: Replica: b7759eaf3a5c40da83ca0902f2d7d6f6, State: Running, Role: LEADER
I20260926 20:58:26.870954 13947 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } }
I20260926 20:58:26.871395 13960 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7759eaf3a5c40da83ca0902f2d7d6f6. Latest consensus state: current_term: 1 leader_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } } }
I20260926 20:58:26.871491 13960 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [sys.catalog]: This master's current role is: LEADER
I20260926 20:58:26.871565 13805 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7759eaf3a5c40da83ca0902f2d7d6f6 in term 1.
I20260926 20:58:26.871733 13962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:58:26.871965 13962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:58:26.873601 13805 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3 [term 1 FOLLOWER]: Refusing update from remote peer b7759eaf3a5c40da83ca0902f2d7d6f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:58:26.873615 13880 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733 [term 1 FOLLOWER]: Refusing update from remote peer b7759eaf3a5c40da83ca0902f2d7d6f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:58:26.873972 13947 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:58:26.874101 13960 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:58:26.875416 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7759eaf3a5c40da83ca0902f2d7d6f6. Latest consensus state: current_term: 1 leader_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } } }
I20260926 20:58:26.875437 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7759eaf3a5c40da83ca0902f2d7d6f6. Latest consensus state: current_term: 1 leader_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } } }
I20260926 20:58:26.875531 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:58:26.875533 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:58:26.876585 13947 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } } }
I20260926 20:58:26.876685 13947 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [sys.catalog]: This master's current role is: LEADER
I20260926 20:58:26.876801 13963 mvcc.cc:205] Tried to move back new op lower bound from 7333709032951353344 to 7333709032944087040. Current Snapshot: MvccSnapshot[applied={T|T < 7333709032951353344}]
I20260926 20:58:26.876791 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } } }
I20260926 20:58:26.876869 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:58:26.876824 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } } }
I20260926 20:58:26.876911 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:58:26.876597 13968 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7759eaf3a5c40da83ca0902f2d7d6f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } peers { permanent_uuid: "66b1993369b34e3888577a3ce1ed30e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40587 } } peers { permanent_uuid: "1196fa070c76435999c32d176a2d8733" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40261 } } }
I20260926 20:58:26.876942 13968 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6 [sys.catalog]: This master's current role is: LEADER
I20260926 20:58:26.876991 13965 mvcc.cc:205] Tried to move back new op lower bound from 7333709032951353344 to 7333709032944087040. Current Snapshot: MvccSnapshot[applied={T|T < 7333709032951353344}]
I20260926 20:58:26.877254 13962 catalog_manager.cc:1347] Generated new cluster ID: 03f8114a1f504aa0894c7a0e07e90ea9
I20260926 20:58:26.877310 13962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:58:26.880651 13964 mvcc.cc:205] Tried to move back new op lower bound from 7333709032951353344 to 7333709032944087040. Current Snapshot: MvccSnapshot[applied={T|T < 7333709032951353344 or (T in {7333709032951353344})}]
I20260926 20:58:26.897785 13962 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:58:26.898438 13962 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:58:26.901686 13962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7759eaf3a5c40da83ca0902f2d7d6f6: Generated new TSK 0
I20260926 20:58:26.901929 13962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:58:27.518620 13923 catalog_manager.cc:1259] Loaded cluster ID: 03f8114a1f504aa0894c7a0e07e90ea9
I20260926 20:58:27.518703 13923 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733: loading cluster ID for follower catalog manager: success
I20260926 20:58:27.519636 13923 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733: acquiring CA information for follower catalog manager: success
I20260926 20:58:27.519985 13923 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1196fa070c76435999c32d176a2d8733: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:58:27.563715 13944 catalog_manager.cc:1259] Loaded cluster ID: 03f8114a1f504aa0894c7a0e07e90ea9
I20260926 20:58:27.563804 13944 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3: loading cluster ID for follower catalog manager: success
I20260926 20:58:27.564746 13944 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3: acquiring CA information for follower catalog manager: success
I20260926 20:58:27.565070 13944 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 66b1993369b34e3888577a3ce1ed30e3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:58:27.959295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 13688
I20260926 20:58:27.965023 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 13763
I20260926 20:58:27.970095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 13838
2026-09-26T20:58:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3161 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:58:28 dist-test-slave-fl7g krb5kdc[13977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:58:28 dist-test-slave-fl7g krb5kdc[13977](info): set up 2 sockets
Sep 26 20:58:28 dist-test-slave-fl7g krb5kdc[13977](info): commencing operation
krb5kdc: starting...
W20260926 20:58:29.016484 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:58:29 dist-test-slave-fl7g krb5kdc[13977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:58:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:58:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:58:29.144367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32997
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:58:29.227577 13990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:29.227806 13990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:29.227830 13990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:29.229068 13990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:58:29.229116 13990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:29.229131 13990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:29.229144 13990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:58:29.229156 13990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:58:29.230610 13990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:29.230829 13990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:29.231011 13990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:29.236025 13996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:29.236044 13998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:29.236047 13995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:29.236342 13990 server_base.cc:1034] running on GCE node
I20260926 20:58:29.236609 13990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:29.236874 13990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:29.238050 13990 hybrid_clock.cc:648] HybridClock initialized: now 1790456309238026 us; error 37 us; skew 500 ppm
Sep 26 20:58:29 dist-test-slave-fl7g krb5kdc[13977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:29.240650 13990 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:58:29.241158 13990 webserver.cc:460] Webserver started at http://127.0.98.62:37671/ using document root <none> and password file <none>
I20260926 20:58:29.241402 13990 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:29.241474 13990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:29.241626 13990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:58:29.242746 13990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "0260bb91cf7543409f724a4457251f42"
format_stamp: "Formatted at 2026-09-26 20:58:29 on dist-test-slave-fl7g"
server_key: "8a24b7bcb22e02dc8d25f6d44d6de8a1"
server_key_iv: "0fbd1cf2cf6f7004647097359885b226"
server_key_version: "encryptionkey@0"
I20260926 20:58:29.243117 13990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "0260bb91cf7543409f724a4457251f42"
format_stamp: "Formatted at 2026-09-26 20:58:29 on dist-test-slave-fl7g"
server_key: "8a24b7bcb22e02dc8d25f6d44d6de8a1"
server_key_iv: "0fbd1cf2cf6f7004647097359885b226"
server_key_version: "encryptionkey@0"
I20260926 20:58:29.244560 13990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:58:29.245361 14005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:29.245530 13990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:58:29.245632 13990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "0260bb91cf7543409f724a4457251f42"
format_stamp: "Formatted at 2026-09-26 20:58:29 on dist-test-slave-fl7g"
server_key: "8a24b7bcb22e02dc8d25f6d44d6de8a1"
server_key_iv: "0fbd1cf2cf6f7004647097359885b226"
server_key_version: "encryptionkey@0"
I20260926 20:58:29.245720 13990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:29.254417 13990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:29.256537 13990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:29.256701 13990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:29.261319 13990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32997
I20260926 20:58:29.261356 14057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32997 every 8 connection(s)
I20260926 20:58:29.261790 13990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:58:29.262693 14058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:29.265280 14058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42: Bootstrap starting.
I20260926 20:58:29.265897 14058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42: Neither blocks nor log segments found. Creating new log.
I20260926 20:58:29.266222 14058 log.cc:824] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:29.266979 14058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42: No bootstrap required, opened a new log
I20260926 20:58:29.268682 14058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0260bb91cf7543409f724a4457251f42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32997 } }
I20260926 20:58:29.268829 14058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:29.268852 14058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0260bb91cf7543409f724a4457251f42, State: Initialized, Role: FOLLOWER
I20260926 20:58:29.269009 14058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0260bb91cf7543409f724a4457251f42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32997 } }
I20260926 20:58:29.269083 14058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:58:29.269140 14058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:58:29.269222 14058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:29.269552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 13990
I20260926 20:58:29.269647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:58:29.269860 392 external_mini_cluster.cc:1428] Setting key a00e9d96980428f6a70fdcfe6747c28b
I20260926 20:58:29.270030 14058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0260bb91cf7543409f724a4457251f42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32997 } }
I20260926 20:58:29.270165 14058 leader_election.cc:302] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0260bb91cf7543409f724a4457251f42; no voters:
I20260926 20:58:29.270360 14058 leader_election.cc:288] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:58:29.270419 14061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:58:29.270601 14061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42 [term 1 LEADER]: Becoming Leader. State: Replica: 0260bb91cf7543409f724a4457251f42, State: Running, Role: LEADER
I20260926 20:58:29.270684 14058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:58:29.270773 14061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0260bb91cf7543409f724a4457251f42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32997 } }
I20260926 20:58:29.271193 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0260bb91cf7543409f724a4457251f42. Latest consensus state: current_term: 1 leader_uuid: "0260bb91cf7543409f724a4457251f42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0260bb91cf7543409f724a4457251f42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32997 } } }
I20260926 20:58:29.271344 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42 [sys.catalog]: This master's current role is: LEADER
I20260926 20:58:29.271839 14071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:58:29.272557 14071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:58:29.272825 14065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0260bb91cf7543409f724a4457251f42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0260bb91cf7543409f724a4457251f42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32997 } } }
I20260926 20:58:29.273104 14065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42 [sys.catalog]: This master's current role is: LEADER
I20260926 20:58:29.274468 14071 catalog_manager.cc:1347] Generated new cluster ID: 72eb0b15c023498cae4fd63de4d3e5cf
I20260926 20:58:29.274551 14071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:58:29 dist-test-slave-fl7g krb5kdc[13977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:29.281939 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:29.270312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.0.98.62:32997)
0926 20:58:29.270489 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:29.270501 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:29.270847 (+ 346us) server_negotiation.cc:407] Connection header received
0926 20:58:29.270977 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:29.270985 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:29.271044 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:29.271184 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:29.272114 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:29.273808 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:29.274882 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:29.275316 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:29.275472 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:29.277942 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:29.277946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:29.277946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:29.277960 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:29.280177 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:29.280661 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:29.280664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:29.280665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:29.280781 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:29.281013 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:29.281013 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:29.281014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:29.281289 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:29.281475 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:29.281548 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 20:58:29.281701 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260926 20:58:29.283366 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:58:29.294194 14071 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:58:29.294800 14071 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:58:29.300452 14071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0260bb91cf7543409f724a4457251f42: Generated new TSK 0
I20260926 20:58:29.300704 14071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:58:29.337962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 13990
2026-09-26T20:58:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1366 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:58:29 dist-test-slave-fl7g krb5kdc[14085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:58:29 dist-test-slave-fl7g krb5kdc[14085](info): set up 2 sockets
Sep 26 20:58:29 dist-test-slave-fl7g krb5kdc[14085](info): commencing operation
krb5kdc: starting...
W20260926 20:58:30.388636 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:58:30 dist-test-slave-fl7g krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:58:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:58:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:58:30.516157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43223
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:58:30.597759 14098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:30.598018 14098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:30.598057 14098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:30.599229 14098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:58:30.599305 14098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:30.599328 14098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:30.599351 14098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:58:30.599380 14098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:58:30.600930 14098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:30.601231 14098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:30.601482 14098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:30.606617 14103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:30.606609 14104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:30.606766 14098 server_base.cc:1034] running on GCE node
W20260926 20:58:30.606612 14106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:30.607054 14098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:30.607321 14098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:30.608512 14098 hybrid_clock.cc:648] HybridClock initialized: now 1790456310608464 us; error 58 us; skew 500 ppm
Sep 26 20:58:30 dist-test-slave-fl7g krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:30.611352 14098 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:58:30.611919 14098 webserver.cc:460] Webserver started at http://127.0.98.62:34973/ using document root <none> and password file <none>
I20260926 20:58:30.612203 14098 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:30.612278 14098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:30.612489 14098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:58:30.613631 14098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "2ecaaac71a38497bb046a4a8234e153e"
format_stamp: "Formatted at 2026-09-26 20:58:30 on dist-test-slave-fl7g"
server_key: "6f1c0a7e937ab89e95da753e77be4b4b"
server_key_iv: "bdf38ca53ef7f04caad9595c1b08c5f0"
server_key_version: "encryptionkey@0"
I20260926 20:58:30.614012 14098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ecaaac71a38497bb046a4a8234e153e"
format_stamp: "Formatted at 2026-09-26 20:58:30 on dist-test-slave-fl7g"
server_key: "6f1c0a7e937ab89e95da753e77be4b4b"
server_key_iv: "bdf38ca53ef7f04caad9595c1b08c5f0"
server_key_version: "encryptionkey@0"
I20260926 20:58:30.615530 14098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:58:30.616474 14113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:30.616675 14098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:30.616801 14098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "2ecaaac71a38497bb046a4a8234e153e"
format_stamp: "Formatted at 2026-09-26 20:58:30 on dist-test-slave-fl7g"
server_key: "6f1c0a7e937ab89e95da753e77be4b4b"
server_key_iv: "bdf38ca53ef7f04caad9595c1b08c5f0"
server_key_version: "encryptionkey@0"
I20260926 20:58:30.616890 14098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:30.636857 14098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:30.639276 14098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:30.639474 14098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:30.644006 14098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43223
I20260926 20:58:30.644074 14165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43223 every 8 connection(s)
I20260926 20:58:30.644506 14098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:58:30.645562 14166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:30.647955 14166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e: Bootstrap starting.
I20260926 20:58:30.648677 14166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e: Neither blocks nor log segments found. Creating new log.
I20260926 20:58:30.648933 14166 log.cc:824] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:30.649658 14166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e: No bootstrap required, opened a new log
I20260926 20:58:30.651342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 14098
I20260926 20:58:30.651458 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:58:30.651412 14166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecaaac71a38497bb046a4a8234e153e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43223 } }
I20260926 20:58:30.651624 14166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:30.651664 14166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecaaac71a38497bb046a4a8234e153e, State: Initialized, Role: FOLLOWER
I20260926 20:58:30.651686 392 external_mini_cluster.cc:1428] Setting key 45362054b95092b4bff05f145d946161
I20260926 20:58:30.651827 14166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecaaac71a38497bb046a4a8234e153e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43223 } }
I20260926 20:58:30.651921 14166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:58:30.651955 14166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:58:30.652031 14166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:30.653203 14166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecaaac71a38497bb046a4a8234e153e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43223 } }
I20260926 20:58:30.653345 14166 leader_election.cc:302] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ecaaac71a38497bb046a4a8234e153e; no voters:
I20260926 20:58:30.653527 14166 leader_election.cc:288] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:58:30.653584 14171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:58:30.653787 14171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 1 LEADER]: Becoming Leader. State: Replica: 2ecaaac71a38497bb046a4a8234e153e, State: Running, Role: LEADER
I20260926 20:58:30.653847 14166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:58:30.653919 14171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecaaac71a38497bb046a4a8234e153e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43223 } }
I20260926 20:58:30.654358 14172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ecaaac71a38497bb046a4a8234e153e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecaaac71a38497bb046a4a8234e153e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43223 } } }
I20260926 20:58:30.654443 14172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [sys.catalog]: This master's current role is: LEADER
I20260926 20:58:30.654649 14173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ecaaac71a38497bb046a4a8234e153e. Latest consensus state: current_term: 1 leader_uuid: "2ecaaac71a38497bb046a4a8234e153e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecaaac71a38497bb046a4a8234e153e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43223 } } }
I20260926 20:58:30.654747 14173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [sys.catalog]: This master's current role is: LEADER
I20260926 20:58:30.655076 14179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:58:30.655658 14179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:58:30.657753 14179 catalog_manager.cc:1347] Generated new cluster ID: 96932c0687ab4665bc2f4647e988aea0
I20260926 20:58:30.657810 14179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:58:30 dist-test-slave-fl7g krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:30.662855 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:30.652254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.0.98.62:43223)
0926 20:58:30.652461 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:30.652471 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:30.652510 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:58:30.652618 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:30.652623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:30.652668 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:30.652789 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:30.653618 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:30.655092 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:30.656090 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:30.656646 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:30.656783 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:30.659323 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:30.659328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:30.659328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:30.659341 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:30.661168 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:30.661570 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:30.661573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:30.661573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:30.661673 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:30.661892 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:30.661893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:30.661893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:30.662153 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:30.662354 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:30.662432 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 20:58:30.662691 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260926 20:58:30.684235 14179 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:58:30.684868 14179 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:58:30.695978 14179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e: Generated new TSK 0
I20260926 20:58:30.696234 14179 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:58:30.731724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43223
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:58:30.835976 14192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:30.836222 14192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:30.836249 14192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:30.837532 14192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:30.837584 14192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:30.837611 14192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:58:30.839102 14192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:30.839320 14192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:30.839516 14192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:30.844688 14198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:30.844684 14200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:30.845093 14197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:30.845172 14192 server_base.cc:1034] running on GCE node
I20260926 20:58:30.845383 14192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:30.845641 14192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:30.846812 14192 hybrid_clock.cc:648] HybridClock initialized: now 1790456310846795 us; error 31 us; skew 500 ppm
Sep 26 20:58:30 dist-test-slave-fl7g krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:30.849574 14192 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:58:30.849992 14192 webserver.cc:460] Webserver started at http://127.0.98.1:38861/ using document root <none> and password file <none>
I20260926 20:58:30.850240 14192 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:30.850323 14192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:30.850482 14192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:58:30.851648 14192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "590bf2a9350f4d51a26c85634269ac39"
format_stamp: "Formatted at 2026-09-26 20:58:30 on dist-test-slave-fl7g"
server_key: "958040b889bbb419296dd5840dba9fe5"
server_key_iv: "874dad09f9aaa11f91c50e6aea07c452"
server_key_version: "encryptionkey@0"
I20260926 20:58:30.852032 14192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "590bf2a9350f4d51a26c85634269ac39"
format_stamp: "Formatted at 2026-09-26 20:58:30 on dist-test-slave-fl7g"
server_key: "958040b889bbb419296dd5840dba9fe5"
server_key_iv: "874dad09f9aaa11f91c50e6aea07c452"
server_key_version: "encryptionkey@0"
I20260926 20:58:30.853560 14192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:58:30.854606 14207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:30.854801 14192 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:58:30.854900 14192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "590bf2a9350f4d51a26c85634269ac39"
format_stamp: "Formatted at 2026-09-26 20:58:30 on dist-test-slave-fl7g"
server_key: "958040b889bbb419296dd5840dba9fe5"
server_key_iv: "874dad09f9aaa11f91c50e6aea07c452"
server_key_version: "encryptionkey@0"
I20260926 20:58:30.854961 14192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:30.872578 14192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:30.875070 14192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:30.875203 14192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:30.875423 14192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:58:30.875794 14192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:58:30.875834 14192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:30.875857 14192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:58:30.875916 14192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:30.882020 14192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38519
I20260926 20:58:30.882114 14320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38519 every 8 connection(s)
I20260926 20:58:30.882436 14192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:58:30.888070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 14192
I20260926 20:58:30.888233 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:58:30.888478 392 external_mini_cluster.cc:1428] Setting key bfaa6a92a3919e330347ffae2790b5cf
Sep 26 20:58:30 dist-test-slave-fl7g krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:30.892656 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:30.883212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43403 (local address 127.0.98.62:43223)
0926 20:58:30.883303 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:30.883305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:30.883482 (+ 177us) server_negotiation.cc:407] Connection header received
0926 20:58:30.883992 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:30.883993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:30.884008 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:30.884082 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:30.885093 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:30.885614 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:30.886680 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:30.886910 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:30.886979 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:30.889383 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:30.889384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:30.889384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:30.889393 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:30.891101 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:30.891626 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:30.891627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:30.891628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:30.891682 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:30.891973 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:30.891973 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:30.891974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:30.892076 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:30.892126 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:30.892510 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 20:58:30.892590 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:58:30.892977 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:30.883174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:43403)
0926 20:58:30.883344 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:30.883382 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:30.883872 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:30.884157 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:30.884161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:30.884784 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:30.884992 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:30.884997 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:30.885720 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:30.885722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:30.886574 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:30.886577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:30.886999 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:30.886999 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:30.887108 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:30.887656 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:30.887665 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:30.889297 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:30.891190 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:30.891200 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:30.891203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:30.891548 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:30.891750 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:30.891751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:30.891751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:30.891909 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:30.892138 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:30.892144 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:30.892330 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 20:58:30.892477 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 20:58:30.892672 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260926 20:58:30.893589 14321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43223
I20260926 20:58:30.893707 14321 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:30.893929 14321 heartbeater.cc:511] Master 127.0.98.62:43223 requested a full tablet report, sending...
I20260926 20:58:30.894495 14130 ts_manager.cc:194] Registered new tserver with Master: 590bf2a9350f4d51a26c85634269ac39 (127.0.98.1:38519)
I20260926 20:58:30.895117 14130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43403
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:58:30.915643 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43223
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:58:31.001240 14326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:31.001469 14326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:31.001494 14326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:31.002607 14326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:31.002645 14326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:31.002671 14326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:58:31.004164 14326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:31.004376 14326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:31.004630 14326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:31.009884 14334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:31.009837 14331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:31.009840 14332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:31.010061 14326 server_base.cc:1034] running on GCE node
I20260926 20:58:31.010422 14326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:31.010665 14326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:31.011864 14326 hybrid_clock.cc:648] HybridClock initialized: now 1790456311011845 us; error 30 us; skew 500 ppm
Sep 26 20:58:31 dist-test-slave-fl7g krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:31.014616 14326 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:58:31.015117 14326 webserver.cc:460] Webserver started at http://127.0.98.2:41755/ using document root <none> and password file <none>
I20260926 20:58:31.015309 14326 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:31.015350 14326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:31.015443 14326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:58:31.016561 14326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "7239e877811f4a348557116462308610"
format_stamp: "Formatted at 2026-09-26 20:58:31 on dist-test-slave-fl7g"
server_key: "4c2fbb8575f918443b6c8f406e3cde4d"
server_key_iv: "67d5ec67a0e70b74aa685783d9bf3a5a"
server_key_version: "encryptionkey@0"
I20260926 20:58:31.016894 14326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7239e877811f4a348557116462308610"
format_stamp: "Formatted at 2026-09-26 20:58:31 on dist-test-slave-fl7g"
server_key: "4c2fbb8575f918443b6c8f406e3cde4d"
server_key_iv: "67d5ec67a0e70b74aa685783d9bf3a5a"
server_key_version: "encryptionkey@0"
I20260926 20:58:31.018272 14326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:31.019101 14341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:31.019330 14326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:31.019423 14326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "7239e877811f4a348557116462308610"
format_stamp: "Formatted at 2026-09-26 20:58:31 on dist-test-slave-fl7g"
server_key: "4c2fbb8575f918443b6c8f406e3cde4d"
server_key_iv: "67d5ec67a0e70b74aa685783d9bf3a5a"
server_key_version: "encryptionkey@0"
I20260926 20:58:31.019515 14326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:31.031153 14326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:31.033445 14326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:31.033576 14326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:31.033787 14326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:58:31.034159 14326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:58:31.034195 14326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:31.034261 14326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:58:31.034289 14326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:31.040227 14326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44125
I20260926 20:58:31.040307 14454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44125 every 8 connection(s)
I20260926 20:58:31.040704 14326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:58:31.040869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 14326
I20260926 20:58:31.040992 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:58:31.041234 392 external_mini_cluster.cc:1428] Setting key 660591af5fd3326e1146a56a4416f467
Sep 26 20:58:31 dist-test-slave-fl7g krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 20:58:31.051666 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.041659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33081 (local address 127.0.98.62:43223)
0926 20:58:31.041710 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:31.041712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:31.041986 (+ 274us) server_negotiation.cc:407] Connection header received
0926 20:58:31.042773 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:31.042775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:31.042784 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:31.042822 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:31.044052 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.044875 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.046038 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.046386 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.046474 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.048875 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:31.048877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:31.048878 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:31.048887 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:31.050420 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:31.050842 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:31.050843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:31.050844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:31.050896 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:31.051125 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:31.051126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:31.051126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:31.051235 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:31.051286 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:31.051525 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 20:58:31.051597 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:58:31.051858 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.041653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:33081)
0926 20:58:31.041825 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:31.041876 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:31.042656 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:31.042898 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:31.042902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:31.043750 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:31.043951 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.043957 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.044977 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.044978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.045934 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.045937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.046553 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.046554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.046669 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.047278 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:31.047287 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:31.048766 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:31.050502 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:31.050508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:31.050510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:31.050742 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:31.050959 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:31.050959 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:31.050960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:31.051070 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:31.051290 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:31.051294 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:31.051416 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 20:58:31.051503 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 20:58:31.051638 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260926 20:58:31.052428 14455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 20:58:31.052541 14455 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:31.052762 14455 heartbeater.cc:511] Master 127.0.98.62:43223 requested a full tablet report, sending...
I20260926 20:58:31.053282 14130 ts_manager.cc:194] Registered new tserver with Master: 7239e877811f4a348557116462308610 (127.0.98.2:44125)
I20260926 20:58:31.053785 14130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:58:31.071277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43223
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:58:31.153874 14460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:31.154206 14460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:31.154249 14460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:31.155668 14460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:31.155718 14460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:31.155748 14460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:58:31.157415 14460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:31.157689 14460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:31.157894 14460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:31.163409 14465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:31.163451 14468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:31.163460 14466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:31.163578 14460 server_base.cc:1034] running on GCE node
I20260926 20:58:31.163890 14460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:31.164175 14460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:31.165372 14460 hybrid_clock.cc:648] HybridClock initialized: now 1790456311165350 us; error 34 us; skew 500 ppm
Sep 26 20:58:31 dist-test-slave-fl7g krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:31.168198 14460 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:58:31.168733 14460 webserver.cc:460] Webserver started at http://127.0.98.3:34089/ using document root <none> and password file <none>
I20260926 20:58:31.168941 14460 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:31.168983 14460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:31.169085 14460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:58:31.170285 14460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "68eea3b3adcc48c48f75c6a58fb9a287"
format_stamp: "Formatted at 2026-09-26 20:58:31 on dist-test-slave-fl7g"
server_key: "69880703044e878e0c0cad46a641aa55"
server_key_iv: "c892406223c96147cb40737ee70fdf17"
server_key_version: "encryptionkey@0"
I20260926 20:58:31.170612 14460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "68eea3b3adcc48c48f75c6a58fb9a287"
format_stamp: "Formatted at 2026-09-26 20:58:31 on dist-test-slave-fl7g"
server_key: "69880703044e878e0c0cad46a641aa55"
server_key_iv: "c892406223c96147cb40737ee70fdf17"
server_key_version: "encryptionkey@0"
I20260926 20:58:31.171962 14460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:58:31.172808 14475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:31.173144 14460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:58:31.173259 14460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "68eea3b3adcc48c48f75c6a58fb9a287"
format_stamp: "Formatted at 2026-09-26 20:58:31 on dist-test-slave-fl7g"
server_key: "69880703044e878e0c0cad46a641aa55"
server_key_iv: "c892406223c96147cb40737ee70fdf17"
server_key_version: "encryptionkey@0"
I20260926 20:58:31.173338 14460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:31.182860 14460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:31.185467 14460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:31.185631 14460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:31.185904 14460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:58:31.186235 14460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:58:31.186290 14460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:31.186333 14460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:58:31.186360 14460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:31.191977 14460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35901
I20260926 20:58:31.192057 14588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35901 every 8 connection(s)
I20260926 20:58:31.192387 14460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:58:31.196666 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 14460
I20260926 20:58:31.196765 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:58:31.196966 392 external_mini_cluster.cc:1428] Setting key 43a22d292e64ada42626876c8c6b807f
Sep 26 20:58:31 dist-test-slave-fl7g krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:31.202373 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.193190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41945 (local address 127.0.98.62:43223)
0926 20:58:31.193267 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:31.193269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:31.193469 (+ 200us) server_negotiation.cc:407] Connection header received
0926 20:58:31.193957 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:31.193958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:31.193967 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:31.194029 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:31.194940 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.195420 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.196909 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.197166 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.197245 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.199595 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:31.199597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:31.199597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:31.199604 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:31.200952 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:31.201355 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:31.201356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:31.201356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:31.201416 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:31.201713 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:31.201714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:31.201715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:31.201838 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:31.201904 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:31.202227 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 20:58:31.202303 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:58:31.202507 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.193157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:41945)
0926 20:58:31.193340 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:31.193383 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:31.193872 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:31.194081 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:31.194085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:31.194666 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:31.194861 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.194865 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.195525 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.195526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.196820 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.196824 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.197296 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.197297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.197451 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.198106 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:31.198114 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:31.199491 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:31.201029 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:31.201034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:31.201036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:31.201264 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:31.201508 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:31.201509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:31.201509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:31.201624 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:31.201927 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:31.201933 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:31.202062 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 20:58:31.202160 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 20:58:31.202288 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":113,"threads_started":1}
I20260926 20:58:31.203066 14589 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43223
I20260926 20:58:31.203181 14589 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:31.203418 14589 heartbeater.cc:511] Master 127.0.98.62:43223 requested a full tablet report, sending...
I20260926 20:58:31.203806 14130 ts_manager.cc:194] Registered new tserver with Master: 68eea3b3adcc48c48f75c6a58fb9a287 (127.0.98.3:35901)
I20260926 20:58:31.204185 14130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41945
I20260926 20:58:31.208986 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:58:31.217350 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.210680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.0.98.62:43223)
0926 20:58:31.210753 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:31.210755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:31.210878 (+ 123us) server_negotiation.cc:407] Connection header received
0926 20:58:31.210937 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:31.210937 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:31.210947 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:31.211004 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:31.211591 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.212071 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.213006 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.213211 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.213285 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.214701 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:31.214703 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:31.214703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:31.214711 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:31.216224 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:31.216654 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:31.216655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:31.216656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:31.216715 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:31.216879 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:31.216880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:31.216880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:31.217042 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:31.217127 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:31.217212 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 20:58:31.217279 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:58:31.218992 14130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:58:31.220985 14130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:58:31.227979 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.223584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38519 (local address 127.0.0.1:34482)
0926 20:58:31.223992 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:31.224002 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:31.224046 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:31.224363 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:31.224366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:31.224369 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:31.224488 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.224491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.225863 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.225864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.226894 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.226897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.227631 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.227632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.227712 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.227727 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 20:58:31.227809 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:58:31.227869 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":68,"thread_start_us":116,"threads_started":1}
I20260926 20:58:31.228101 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.223580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44125 (local address 127.0.0.1:56886)
0926 20:58:31.223756 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:31.223801 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:31.223873 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:31.224390 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:31.224391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:31.224394 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:31.224498 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.224501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.225944 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.225945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.226891 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.226894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.227817 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.227818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.227872 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.227875 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:58:31.227944 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:58:31.227976 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":54,"thread_start_us":64,"threads_started":1}
I20260926 20:58:31.228379 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.223811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.0.98.1:38519)
0926 20:58:31.224162 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:31.224167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:31.224178 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:58:31.224218 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:31.224220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:31.224240 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:31.224323 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:31.224623 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.225750 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.227013 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.227518 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.227628 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.227697 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:31.228224 (+ 527us) server_negotiation.cc:299] Negotiation successful
0926 20:58:31.228280 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":86,"threads_started":1}
I20260926 20:58:31.228641 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.223828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.0.98.2:44125)
0926 20:58:31.224219 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:31.224224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:31.224234 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:58:31.224279 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:31.224281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:31.224298 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:31.224380 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:31.224587 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.225750 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.226986 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.227725 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.227799 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.227848 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:31.228468 (+ 620us) server_negotiation.cc:299] Negotiation successful
0926 20:58:31.228525 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":77,"threads_started":1}
I20260926 20:58:31.229180 14255 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0508815d31431eb5414db2d5784395 (DEFAULT_TABLE table=test-table [id=85196e473ede469fa13e065f421365cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:58:31.229544 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.223592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35901 (local address 127.0.0.1:46520)
0926 20:58:31.224006 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:31.224018 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:31.224079 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:31.224701 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:31.224702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:31.224704 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:31.224741 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.224743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.225725 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.225726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.227906 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.227910 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.229299 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.229300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.229368 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.229372 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:58:31.229420 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 20:58:31.229466 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":971,"thread_start_us":135,"threads_started":1}
I20260926 20:58:31.229568 14255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f0508815d31431eb5414db2d5784395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:31.229516 14389 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0508815d31431eb5414db2d5784395 (DEFAULT_TABLE table=test-table [id=85196e473ede469fa13e065f421365cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:58:31.229861 14389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f0508815d31431eb5414db2d5784395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:31.229867 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.223857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46520 (local address 127.0.98.3:35901)
0926 20:58:31.224534 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:31.224538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:31.224548 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:58:31.224583 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:31.224585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:31.224599 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:31.224867 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:31.224933 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.225597 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.227988 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.229144 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.229215 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.229265 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:31.229734 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 20:58:31.229780 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":88,"threads_started":1}
I20260926 20:58:31.230461 14523 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0508815d31431eb5414db2d5784395 (DEFAULT_TABLE table=test-table [id=85196e473ede469fa13e065f421365cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:58:31.230767 14523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f0508815d31431eb5414db2d5784395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:31.232661 14606 tablet_bootstrap.cc:492] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39: Bootstrap starting.
I20260926 20:58:31.233251 14607 tablet_bootstrap.cc:492] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610: Bootstrap starting.
I20260926 20:58:31.233349 14606 tablet_bootstrap.cc:654] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39: Neither blocks nor log segments found. Creating new log.
I20260926 20:58:31.233959 14606 log.cc:824] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:31.233968 14608 tablet_bootstrap.cc:492] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287: Bootstrap starting.
I20260926 20:58:31.234037 14607 tablet_bootstrap.cc:654] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610: Neither blocks nor log segments found. Creating new log.
I20260926 20:58:31.234596 14607 log.cc:824] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:31.234756 14608 tablet_bootstrap.cc:654] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287: Neither blocks nor log segments found. Creating new log.
I20260926 20:58:31.235033 14606 tablet_bootstrap.cc:492] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39: No bootstrap required, opened a new log
I20260926 20:58:31.235172 14606 ts_tablet_manager.cc:1397] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 20:58:31.235304 14608 log.cc:824] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:31.235436 14607 tablet_bootstrap.cc:492] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610: No bootstrap required, opened a new log
I20260926 20:58:31.235508 14607 ts_tablet_manager.cc:1397] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:58:31.236111 14608 tablet_bootstrap.cc:492] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287: No bootstrap required, opened a new log
I20260926 20:58:31.236186 14608 ts_tablet_manager.cc:1397] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:58:31.237316 14606 raft_consensus.cc:348] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:31.237375 14607 raft_consensus.cc:348] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:31.237496 14607 raft_consensus.cc:374] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:31.237623 14607 raft_consensus.cc:729] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7239e877811f4a348557116462308610, State: Initialized, Role: FOLLOWER
I20260926 20:58:31.237536 14606 raft_consensus.cc:374] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:31.237764 14607 consensus_queue.cc:260] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:31.237823 14606 raft_consensus.cc:729] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590bf2a9350f4d51a26c85634269ac39, State: Initialized, Role: FOLLOWER
I20260926 20:58:31.237943 14606 consensus_queue.cc:260] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:31.238095 14607 ts_tablet_manager.cc:1428] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:58:31.238204 14455 heartbeater.cc:503] Master 127.0.98.62:43223 was elected leader, sending a full tablet report...
I20260926 20:58:31.238269 14606 ts_tablet_manager.cc:1428] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:58:31.238392 14321 heartbeater.cc:503] Master 127.0.98.62:43223 was elected leader, sending a full tablet report...
I20260926 20:58:31.238341 14608 raft_consensus.cc:348] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:31.238781 14608 raft_consensus.cc:374] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:31.238862 14608 raft_consensus.cc:729] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68eea3b3adcc48c48f75c6a58fb9a287, State: Initialized, Role: FOLLOWER
I20260926 20:58:31.239012 14608 consensus_queue.cc:260] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:31.239506 14589 heartbeater.cc:503] Master 127.0.98.62:43223 was elected leader, sending a full tablet report...
I20260926 20:58:31.239516 14608 ts_tablet_manager.cc:1428] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260926 20:58:31.291600 14456 tablet.cc:2369] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:58:31.327450 14612 raft_consensus.cc:482] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:58:31.327584 14612 raft_consensus.cc:504] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:31.328047 14612 leader_election.cc:288] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68eea3b3adcc48c48f75c6a58fb9a287 (127.0.98.3:35901), 590bf2a9350f4d51a26c85634269ac39 (127.0.98.1:38519)
I20260926 20:58:31.330871 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.328330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50473 (local address 127.0.98.1:38519)
0926 20:58:31.328373 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:31.328375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:31.328438 (+ 63us) server_negotiation.cc:407] Connection header received
0926 20:58:31.328470 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:31.328471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:31.328485 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:31.328535 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:31.328700 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.329195 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.330124 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.330503 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.330586 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.330599 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:31.330758 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 20:58:31.330800 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:58:31.330871 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.328247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38519 (local address 127.0.98.2:50473)
0926 20:58:31.328299 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:31.328313 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:31.328367 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:31.328539 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:31.328539 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:31.328541 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:31.328621 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.328622 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.329288 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.329289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.330025 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.330027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.330599 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.330601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.330680 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.330683 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:58:31.330752 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:58:31.330796 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 20:58:31.331002 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.328401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39049 (local address 127.0.98.3:35901)
0926 20:58:31.328476 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:31.328478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:31.328600 (+ 122us) server_negotiation.cc:407] Connection header received
0926 20:58:31.328666 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:31.328667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:31.328680 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:31.328755 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:31.328914 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.329371 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.330292 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.330661 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.330745 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.330755 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:31.330900 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 20:58:31.330938 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:58:31.331014 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.328274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35901 (local address 127.0.98.2:39049)
0926 20:58:31.328515 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:31.328523 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:31.328605 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:31.328758 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:31.328759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:31.328761 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:31.328838 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.328839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.329458 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.329459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.330201 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.330203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.330747 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.330748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.330825 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.330829 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:58:31.330894 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:58:31.330933 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":113,"threads_started":1}
I20260926 20:58:31.331353 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0508815d31431eb5414db2d5784395" candidate_uuid: "7239e877811f4a348557116462308610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" is_pre_election: true
I20260926 20:58:31.331403 14275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0508815d31431eb5414db2d5784395" candidate_uuid: "7239e877811f4a348557116462308610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "590bf2a9350f4d51a26c85634269ac39" is_pre_election: true
I20260926 20:58:31.331529 14543 raft_consensus.cc:2454] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7239e877811f4a348557116462308610 in term 0.
I20260926 20:58:31.331568 14275 raft_consensus.cc:2454] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7239e877811f4a348557116462308610 in term 0.
I20260926 20:58:31.331843 14344 leader_election.cc:302] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68eea3b3adcc48c48f75c6a58fb9a287, 7239e877811f4a348557116462308610; no voters:
I20260926 20:58:31.332031 14612 raft_consensus.cc:2781] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:58:31.332070 14612 raft_consensus.cc:482] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:58:31.332086 14612 raft_consensus.cc:3037] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:31.332809 14612 raft_consensus.cc:504] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:31.332958 14612 leader_election.cc:288] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [CANDIDATE]: Term 1 election: Requested vote from peers 68eea3b3adcc48c48f75c6a58fb9a287 (127.0.98.3:35901), 590bf2a9350f4d51a26c85634269ac39 (127.0.98.1:38519)
I20260926 20:58:31.333124 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0508815d31431eb5414db2d5784395" candidate_uuid: "7239e877811f4a348557116462308610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68eea3b3adcc48c48f75c6a58fb9a287"
I20260926 20:58:31.333112 14275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0508815d31431eb5414db2d5784395" candidate_uuid: "7239e877811f4a348557116462308610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "590bf2a9350f4d51a26c85634269ac39"
I20260926 20:58:31.333204 14543 raft_consensus.cc:3037] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:31.333225 14275 raft_consensus.cc:3037] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:31.333907 14543 raft_consensus.cc:2454] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7239e877811f4a348557116462308610 in term 1.
I20260926 20:58:31.334053 14275 raft_consensus.cc:2454] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7239e877811f4a348557116462308610 in term 1.
I20260926 20:58:31.334069 14344 leader_election.cc:302] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68eea3b3adcc48c48f75c6a58fb9a287, 7239e877811f4a348557116462308610; no voters:
I20260926 20:58:31.334205 14612 raft_consensus.cc:2781] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:58:31.334398 14612 raft_consensus.cc:686] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [term 1 LEADER]: Becoming Leader. State: Replica: 7239e877811f4a348557116462308610, State: Running, Role: LEADER
I20260926 20:58:31.334508 14612 consensus_queue.cc:237] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:31.335328 14130 catalog_manager.cc:5495] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7239e877811f4a348557116462308610 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7239e877811f4a348557116462308610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } health_report { overall_health: UNKNOWN } } }
I20260926 20:58:31.346449 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.344158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56888 (local address 127.0.98.2:44125)
0926 20:58:31.344236 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:31.344242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:31.344250 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:58:31.344284 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:31.344284 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:31.344303 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:31.344361 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:58:31.344552 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.345025 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.345736 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.345912 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.345962 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.346096 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:58:31.346193 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:58:31.346260 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:31.346299 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 20:58:31.346316 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:58:31.348356 14275 raft_consensus.cc:1261] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Refusing update from remote peer 7239e877811f4a348557116462308610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:58:31.348357 14543 raft_consensus.cc:1261] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287 [term 1 FOLLOWER]: Refusing update from remote peer 7239e877811f4a348557116462308610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:58:31.348768 14612 consensus_queue.cc:1035] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:58:31.348922 14616 consensus_queue.cc:1035] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:58:31.353737 14130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:58:31.353849 14130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:58:31.354619 14130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e: Sending DeleteTablet for 3 replicas of tablet 2f0508815d31431eb5414db2d5784395
I20260926 20:58:31.355002 14255 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0508815d31431eb5414db2d5784395 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:58:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34482
I20260926 20:58:31.355013 14523 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0508815d31431eb5414db2d5784395 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:58:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46520
I20260926 20:58:31.355075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 14192
I20260926 20:58:31.355232 14389 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0508815d31431eb5414db2d5784395 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:58:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56886
I20260926 20:58:31.356288 14630 tablet_replica.cc:330] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287: stopping tablet replica
I20260926 20:58:31.356428 14630 raft_consensus.cc:2229] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:58:31.356546 14630 raft_consensus.cc:2258] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:58:31.356989 14630 ts_tablet_manager.cc:1905] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:58:31.358666 14630 ts_tablet_manager.cc:1918] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:58:31.358786 14630 log.cc:1196] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/2f0508815d31431eb5414db2d5784395
I20260926 20:58:31.359061 14630 ts_tablet_manager.cc:1939] T 2f0508815d31431eb5414db2d5784395 P 68eea3b3adcc48c48f75c6a58fb9a287: Deleting consensus metadata
I20260926 20:58:31.359510 14116 catalog_manager.cc:4841] TS 68eea3b3adcc48c48f75c6a58fb9a287 (127.0.98.3:35901): tablet 2f0508815d31431eb5414db2d5784395 (table test-table [id=85196e473ede469fa13e065f421365cd]) successfully deleted
I20260926 20:58:31.360549 14631 tablet_replica.cc:330] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610: stopping tablet replica
I20260926 20:58:31.360715 14631 raft_consensus.cc:2229] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [term 1 LEADER]: Raft consensus shutting down.
W20260926 20:58:31.360831 14115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38519 (error 108)
I20260926 20:58:31.360849 14631 raft_consensus.cc:2258] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:58:31.360886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 14326
I20260926 20:58:31.362491 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.361872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38519 (local address 127.0.0.1:34498)
0926 20:58:31.362415 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:31.362460 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520}
W20260926 20:58:31.362636 14115 catalog_manager.cc:4584] TS 590bf2a9350f4d51a26c85634269ac39 (127.0.98.1:38519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f0508815d31431eb5414db2d5784395: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38519: connect: Connection refused (error 111)
I20260926 20:58:31.366844 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.366718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44125 (local address 127.0.0.1:56892)
0926 20:58:31.366781 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:31.366814 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 20:58:31.366885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 14460
I20260926 20:58:31.371521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 14098
I20260926 20:58:31.377487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43223
--webserver_interface=127.0.98.62
--webserver_port=34973
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:58:31.461385 14633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:31.461609 14633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:31.461635 14633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:31.462913 14633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:58:31.462961 14633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:31.462975 14633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:31.462989 14633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:58:31.463042 14633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:58:31.464704 14633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:31.464974 14633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:31.465229 14633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:31.473955 14640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:31.473958 14642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:31.474092 14633 server_base.cc:1034] running on GCE node
W20260926 20:58:31.473958 14639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:31.474488 14633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:31.474867 14633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:31.476074 14633 hybrid_clock.cc:648] HybridClock initialized: now 1790456311476064 us; error 48 us; skew 500 ppm
Sep 26 20:58:31 dist-test-slave-fl7g krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:31.478886 14633 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:58:31.479312 14633 webserver.cc:460] Webserver started at http://127.0.98.62:34973/ using document root <none> and password file <none>
I20260926 20:58:31.479522 14633 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:31.479574 14633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:31.481027 14633 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:58:31.481671 14649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:31.481904 14633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:58:31.482018 14633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "2ecaaac71a38497bb046a4a8234e153e"
format_stamp: "Formatted at 2026-09-26 20:58:30 on dist-test-slave-fl7g"
server_key: "6f1c0a7e937ab89e95da753e77be4b4b"
server_key_iv: "bdf38ca53ef7f04caad9595c1b08c5f0"
server_key_version: "encryptionkey@0"
I20260926 20:58:31.482456 14633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:31.506225 14633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:31.508781 14633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:31.509002 14633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:31.513278 14633 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43223
I20260926 20:58:31.513331 14701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43223 every 8 connection(s)
I20260926 20:58:31.513725 14633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:58:31.515412 14702 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:58:31.516055 14702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e: Bootstrap starting.
I20260926 20:58:31.523000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 14633
I20260926 20:58:31.523370 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38519
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43223
--webserver_port=38861
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:58:31.525058 14702 log.cc:824] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:31.527246 14702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:58:31.527549 14702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e: Bootstrap complete.
I20260926 20:58:31.529989 14702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecaaac71a38497bb046a4a8234e153e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43223 } }
I20260926 20:58:31.530248 14702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecaaac71a38497bb046a4a8234e153e, State: Initialized, Role: FOLLOWER
I20260926 20:58:31.530427 14702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecaaac71a38497bb046a4a8234e153e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43223 } }
I20260926 20:58:31.530499 14702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:58:31.530555 14702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:58:31.530604 14702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:58:31.531668 14702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecaaac71a38497bb046a4a8234e153e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43223 } }
I20260926 20:58:31.531766 14702 leader_election.cc:302] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ecaaac71a38497bb046a4a8234e153e; no voters:
I20260926 20:58:31.531991 14702 leader_election.cc:288] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:58:31.532066 14706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:58:31.532291 14702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:58:31.532312 14706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 2 LEADER]: Becoming Leader. State: Replica: 2ecaaac71a38497bb046a4a8234e153e, State: Running, Role: LEADER
I20260926 20:58:31.532567 14706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecaaac71a38497bb046a4a8234e153e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43223 } }
I20260926 20:58:31.532924 14707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2ecaaac71a38497bb046a4a8234e153e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecaaac71a38497bb046a4a8234e153e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43223 } } }
I20260926 20:58:31.532945 14708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ecaaac71a38497bb046a4a8234e153e. Latest consensus state: current_term: 2 leader_uuid: "2ecaaac71a38497bb046a4a8234e153e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecaaac71a38497bb046a4a8234e153e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43223 } } }
I20260926 20:58:31.533042 14707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [sys.catalog]: This master's current role is: LEADER
I20260926 20:58:31.533046 14708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [sys.catalog]: This master's current role is: LEADER
I20260926 20:58:31.533250 14715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:58:31.534129 14715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:58:31.534370 14715 catalog_manager.cc:1259] Loaded cluster ID: 96932c0687ab4665bc2f4647e988aea0
I20260926 20:58:31.534539 14715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:58:31.534878 14715 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 20:58:31.539651 14715 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 20:58:31.542425 14715 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:58:31.542655 14715 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e: Loaded TSK: 0
I20260926 20:58:31.542894 14715 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:58:31.611475 14704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:31.611729 14704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:31.611759 14704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:31.613289 14704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:31.613364 14704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:31.613401 14704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:58:31.615335 14704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38861
--tserver_master_addrs=127.0.98.62:43223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:31.615621 14704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:31.615895 14704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:31.622315 14732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:31.622315 14729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:31.622481 14704 server_base.cc:1034] running on GCE node
W20260926 20:58:31.622330 14730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:31.622869 14704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:31.623333 14704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:31.624500 14704 hybrid_clock.cc:648] HybridClock initialized: now 1790456311624466 us; error 44 us; skew 500 ppm
Sep 26 20:58:31 dist-test-slave-fl7g krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:31.627313 14704 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:58:31.627854 14704 webserver.cc:460] Webserver started at http://127.0.98.1:38861/ using document root <none> and password file <none>
I20260926 20:58:31.628106 14704 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:31.628222 14704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:31.629940 14704 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:31.630717 14739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:31.630976 14704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:31.631098 14704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "590bf2a9350f4d51a26c85634269ac39"
format_stamp: "Formatted at 2026-09-26 20:58:30 on dist-test-slave-fl7g"
server_key: "958040b889bbb419296dd5840dba9fe5"
server_key_iv: "874dad09f9aaa11f91c50e6aea07c452"
server_key_version: "encryptionkey@0"
I20260926 20:58:31.631501 14704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:31.649317 14704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:31.652503 14704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:31.652706 14704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:31.652989 14704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:58:31.653546 14746 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:58:31.655117 14704 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:58:31.655180 14704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 20:58:31.655220 14704 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:58:31.655941 14704 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:58:31.655988 14704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 20:58:31.656044 14746 tablet_bootstrap.cc:492] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39: Bootstrap starting.
I20260926 20:58:31.664496 14704 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38519
I20260926 20:58:31.664525 14853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38519 every 8 connection(s)
I20260926 20:58:31.665163 14704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:58:31.668169 14746 log.cc:824] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:31.669365 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 14704
I20260926 20:58:31.669668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44125
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43223
--webserver_port=41755
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:58:31.671470 14746 tablet_bootstrap.cc:492] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:58:31.671883 14746 tablet_bootstrap.cc:492] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39: Bootstrap complete.
I20260926 20:58:31.673537 14746 ts_tablet_manager.cc:1397] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.014s
Sep 26 20:58:31 dist-test-slave-fl7g krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:31.674932 14746 raft_consensus.cc:348] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:31.675266 14746 raft_consensus.cc:729] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590bf2a9350f4d51a26c85634269ac39, State: Initialized, Role: FOLLOWER
I20260926 20:58:31.675473 14746 consensus_queue.cc:260] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:31.676111 14746 ts_tablet_manager.cc:1428] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 20:58:31.679804 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.666038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:60159)
0926 20:58:31.666405 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:31.666453 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:31.667100 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:31.667719 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:31.667725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:31.668121 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:31.668396 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.668444 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.670124 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.670125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.671509 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.671512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.672425 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.672427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.672557 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.673215 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:31.673226 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:31.675320 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:31.677908 (+ 2588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:31.677919 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:31.677922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:31.678209 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:31.678541 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:31.678541 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:31.678542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:31.678659 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:31.679171 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:31.679176 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:31.679320 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 20:58:31.679409 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:58:31.679620 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":95,"threads_started":1}
I20260926 20:58:31.679924 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.666117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60159 (local address 127.0.98.62:43223)
0926 20:58:31.666314 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:31.666324 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:31.666624 (+ 300us) server_negotiation.cc:407] Connection header received
0926 20:58:31.667399 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:31.667410 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:31.667493 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:31.667674 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:31.668578 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.669983 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.671620 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.672160 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.672247 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.675478 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:31.675483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:31.675484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:31.675499 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:31.677787 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:31.678303 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:31.678307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:31.678308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:31.678459 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:31.678732 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:31.678733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:31.678734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:31.679093 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:31.679195 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:31.679507 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 20:58:31.679674 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":84,"threads_started":1}
I20260926 20:58:31.680642 14854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43223
I20260926 20:58:31.680768 14854 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:31.681008 14854 heartbeater.cc:511] Master 127.0.98.62:43223 requested a full tablet report, sending...
I20260926 20:58:31.681834 14666 ts_manager.cc:194] Registered new tserver with Master: 590bf2a9350f4d51a26c85634269ac39 (127.0.98.1:38519)
I20260926 20:58:31.683075 14666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60159
I20260926 20:58:31.683974 14854 heartbeater.cc:503] Master 127.0.98.62:43223 was elected leader, sending a full tablet report...
Sep 26 20:58:31 dist-test-slave-fl7g krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:58:31.692219 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.682483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38519 (local address 127.0.0.1:34504)
0926 20:58:31.682736 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:31.682771 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:31.682845 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:31.683052 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:31.683057 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:31.683405 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:31.683471 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.683475 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.684889 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.684890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.685485 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.685487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.686126 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.686129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.686194 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.686773 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:31.686777 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:31.688564 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:31.690841 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:31.690852 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:31.690855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:31.691137 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:31.691397 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:31.691397 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:31.691398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:31.691464 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:31.691832 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:31.691839 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:31.691898 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 20:58:31.691981 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 20:58:31.692142 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":115,"threads_started":1}
I20260926 20:58:31.692323 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.682578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.0.98.1:38519)
0926 20:58:31.682758 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:31.682763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:31.682826 (+ 63us) server_negotiation.cc:407] Connection header received
0926 20:58:31.682940 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:31.682942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:31.682960 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:31.683038 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:31.683548 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.684804 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.685568 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.686039 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.686108 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.688677 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:31.688690 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:31.688690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:31.688699 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:31.690719 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:31.691234 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:31.691237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:31.691238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:31.691320 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:31.691529 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:31.691530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:31.691530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:31.691738 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:31.691825 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:31.692048 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 20:58:31.692170 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
I20260926 20:58:31.693043 14788 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0508815d31431eb5414db2d5784395 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:58:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34504
I20260926 20:58:31.693677 14863 tablet_replica.cc:330] stopping tablet replica
I20260926 20:58:31.693794 14863 raft_consensus.cc:2229] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:58:31.693866 14863 raft_consensus.cc:2258] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:58:31.694198 14863 ts_tablet_manager.cc:1905] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:58:31.696734 14863 ts_tablet_manager.cc:1918] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:58:31.696825 14863 log.cc:1196] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/wal/wals/2f0508815d31431eb5414db2d5784395
I20260926 20:58:31.697049 14863 ts_tablet_manager.cc:1939] T 2f0508815d31431eb5414db2d5784395 P 590bf2a9350f4d51a26c85634269ac39: Deleting consensus metadata
I20260926 20:58:31.697901 14651 catalog_manager.cc:4841] TS 590bf2a9350f4d51a26c85634269ac39 (127.0.98.1:38519): tablet 2f0508815d31431eb5414db2d5784395 (table test-table [id=85196e473ede469fa13e065f421365cd]) successfully deleted
W20260926 20:58:31.764883 14859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:31.765147 14859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:31.765189 14859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:31.766403 14859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:31.766489 14859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:31.766572 14859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:58:31.768182 14859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41755
--tserver_master_addrs=127.0.98.62:43223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:31.768519 14859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:31.768791 14859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:31.774391 14868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:31.774391 14871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:31.774446 14869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:31.774672 14859 server_base.cc:1034] running on GCE node
I20260926 20:58:31.774911 14859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:31.775146 14859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:31.776310 14859 hybrid_clock.cc:648] HybridClock initialized: now 1790456311776299 us; error 32 us; skew 500 ppm
Sep 26 20:58:31 dist-test-slave-fl7g krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:31.779142 14859 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:58:31.779692 14859 webserver.cc:460] Webserver started at http://127.0.98.2:41755/ using document root <none> and password file <none>
I20260926 20:58:31.779949 14859 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:31.780026 14859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:31.781617 14859 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:31.782322 14878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:31.782589 14859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:31.782696 14859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "7239e877811f4a348557116462308610"
format_stamp: "Formatted at 2026-09-26 20:58:31 on dist-test-slave-fl7g"
server_key: "4c2fbb8575f918443b6c8f406e3cde4d"
server_key_iv: "67d5ec67a0e70b74aa685783d9bf3a5a"
server_key_version: "encryptionkey@0"
I20260926 20:58:31.783077 14859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:31.795514 14859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:31.798061 14859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:31.798195 14859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:31.798497 14859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:58:31.798995 14885 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:58:31.800583 14859 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:58:31.800659 14859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 20:58:31.800701 14859 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:58:31.801366 14859 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:58:31.801424 14859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 20:58:31.801510 14885 tablet_bootstrap.cc:492] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610: Bootstrap starting.
I20260926 20:58:31.809060 14859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44125
I20260926 20:58:31.809156 14992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44125 every 8 connection(s)
I20260926 20:58:31.809507 14859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:58:31.809818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 14859
I20260926 20:58:31.810159 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35901
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43223
--webserver_port=34089
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:58:31.812484 14885 log.cc:824] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:31.815385 14885 tablet_bootstrap.cc:492] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:58:31.815712 14885 tablet_bootstrap.cc:492] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610: Bootstrap complete.
I20260926 20:58:31.816922 14885 ts_tablet_manager.cc:1397] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.010s
I20260926 20:58:31.818459 14885 raft_consensus.cc:348] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:31.818763 14885 raft_consensus.cc:729] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7239e877811f4a348557116462308610, State: Initialized, Role: FOLLOWER
I20260926 20:58:31.818913 14885 consensus_queue.cc:260] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
Sep 26 20:58:31 dist-test-slave-fl7g krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:31.819263 14885 ts_tablet_manager.cc:1428] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260926 20:58:31.823007 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.810715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45789 (local address 127.0.98.62:43223)
0926 20:58:31.810800 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:31.810803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:31.812311 (+ 1508us) server_negotiation.cc:407] Connection header received
0926 20:58:31.813017 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:31.813018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:31.813036 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:31.813102 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:31.813953 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.814702 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.816293 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.816708 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.816806 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.819686 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:31.819688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:31.819688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:31.819699 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:31.821586 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:31.822026 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:31.822027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:31.822027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:31.822092 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:31.822324 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:31.822324 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:31.822325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:31.822469 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:31.822536 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:31.822810 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 20:58:31.822905 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:58:31.823056 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.810418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:45789)
0926 20:58:31.812136 (+ 1718us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:31.812188 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:31.812889 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:31.813167 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:31.813172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:31.813528 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:31.813823 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.813830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.814847 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.814848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.816182 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.816187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.816814 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.816815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.816984 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.817730 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:31.817741 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:31.819573 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:31.821697 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:31.821705 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:31.821707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:31.821950 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:31.822159 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:31.822160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:31.822160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:31.822265 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:31.822532 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:31.822536 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:31.822661 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 20:58:31.822772 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 20:58:31.822906 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1674,"thread_start_us":89,"threads_started":1}
I20260926 20:58:31.823835 14993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43223
I20260926 20:58:31.823926 14993 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:31.824154 14993 heartbeater.cc:511] Master 127.0.98.62:43223 requested a full tablet report, sending...
I20260926 20:58:31.824793 14666 ts_manager.cc:194] Registered new tserver with Master: 7239e877811f4a348557116462308610 (127.0.98.2:44125)
I20260926 20:58:31.825387 14666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45789
I20260926 20:58:31.826224 14993 heartbeater.cc:503] Master 127.0.98.62:43223 was elected leader, sending a full tablet report...
Sep 26 20:58:31 dist-test-slave-fl7g krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 20:58:31.833714 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.825103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44125 (local address 127.0.0.1:56902)
0926 20:58:31.825150 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:31.825160 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:31.825203 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:31.825886 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:31.825887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:31.826049 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:31.826099 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.826101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.827051 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.827052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.827973 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.827977 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.828534 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.828537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.828613 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.829047 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:31.829052 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:31.830389 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:31.832475 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:31.832477 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:31.832478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:31.832692 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:31.832944 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:31.832945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:31.832946 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:31.833011 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:31.833355 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:31.833356 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:31.833398 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 20:58:31.833454 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:58:31.833608 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 20:58:31.833716 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.825535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.0.98.2:44125)
0926 20:58:31.825731 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:31.825736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:31.825745 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:58:31.825777 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:31.825779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:31.825795 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:31.825867 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:31.826165 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.826983 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.828086 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.828385 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.828538 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.830476 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:31.830485 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:31.830485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:31.830493 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:31.832314 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:31.832775 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:31.832779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:31.832779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:31.832849 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:31.833082 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:31.833082 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:31.833083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:31.833284 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:31.833369 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:31.833489 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 20:58:31.833603 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":88,"threads_started":1}
I20260926 20:58:31.834506 14927 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0508815d31431eb5414db2d5784395 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:58:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56902
I20260926 20:58:31.835175 15000 tablet_replica.cc:330] stopping tablet replica
I20260926 20:58:31.835315 15000 raft_consensus.cc:2229] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:58:31.835395 15000 raft_consensus.cc:2258] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:58:31.835712 15000 ts_tablet_manager.cc:1905] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:58:31.837944 15000 ts_tablet_manager.cc:1918] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:58:31.838019 15000 log.cc:1196] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/2f0508815d31431eb5414db2d5784395
I20260926 20:58:31.838256 15000 ts_tablet_manager.cc:1939] T 2f0508815d31431eb5414db2d5784395 P 7239e877811f4a348557116462308610: Deleting consensus metadata
I20260926 20:58:31.838753 14650 catalog_manager.cc:4841] TS 7239e877811f4a348557116462308610 (127.0.98.2:44125): tablet 2f0508815d31431eb5414db2d5784395 (table test-table [id=85196e473ede469fa13e065f421365cd]) successfully deleted
W20260926 20:58:31.900317 14996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:31.900674 14996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:31.900724 14996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:31.901978 14996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:31.902058 14996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:31.902107 14996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:58:31.903657 14996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34089
--tserver_master_addrs=127.0.98.62:43223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:31.903966 14996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:31.904225 14996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:31.909583 15005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:31.909569 15008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:31.909660 14996 server_base.cc:1034] running on GCE node
W20260926 20:58:31.909569 15006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:31.909986 14996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:31.910370 14996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:31.911541 14996 hybrid_clock.cc:648] HybridClock initialized: now 1790456311911531 us; error 33 us; skew 500 ppm
Sep 26 20:58:31 dist-test-slave-fl7g krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:31.914240 14996 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:58:31.914731 14996 webserver.cc:460] Webserver started at http://127.0.98.3:34089/ using document root <none> and password file <none>
I20260926 20:58:31.914978 14996 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:31.915058 14996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:31.916556 14996 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:31.917198 15015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:31.917388 14996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:58:31.917483 14996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "68eea3b3adcc48c48f75c6a58fb9a287"
format_stamp: "Formatted at 2026-09-26 20:58:31 on dist-test-slave-fl7g"
server_key: "69880703044e878e0c0cad46a641aa55"
server_key_iv: "c892406223c96147cb40737ee70fdf17"
server_key_version: "encryptionkey@0"
I20260926 20:58:31.917852 14996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:31.939163 14996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:31.941527 14996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:31.941737 14996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:31.942013 14996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:58:31.942371 14996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:58:31.942430 14996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:31.942472 14996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:58:31.942531 14996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:31.949365 14996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35901
I20260926 20:58:31.949409 15128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35901 every 8 connection(s)
I20260926 20:58:31.949815 14996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:58:31.956255 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 14996
Sep 26 20:58:31 dist-test-slave-fl7g krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:31.961545 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.950597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:51091)
0926 20:58:31.950807 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:31.950849 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:31.951666 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:31.952008 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:31.952013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:31.952803 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:31.953090 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.953096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.953834 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.953835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.954685 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.954687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.955103 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.955104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.955255 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.955905 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:31.955914 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:31.957694 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:31.959791 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:31.959800 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:31.959804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:31.960146 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:31.960357 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:31.960358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:31.960359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:31.960524 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:31.960817 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:31.960823 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:31.960998 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 20:58:31.961096 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 20:58:31.961266 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
I20260926 20:58:31.961931 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.950613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51091 (local address 127.0.98.62:43223)
0926 20:58:31.950703 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:31.950705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:31.950945 (+ 240us) server_negotiation.cc:407] Connection header received
0926 20:58:31.951830 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:31.951832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:31.951848 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:31.951934 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:31.953205 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.953724 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.954789 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.955002 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.955093 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.957805 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:31.957807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:31.957808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:31.957817 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:31.959702 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:31.960231 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:31.960233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:31.960233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:31.960302 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:31.960599 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:31.960600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:31.960600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:31.960728 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:31.960789 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:31.961756 (+ 967us) server_negotiation.cc:299] Negotiation successful
0926 20:58:31.961840 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:58:31.962401 15129 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43223
I20260926 20:58:31.962530 15129 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:31.962774 15129 heartbeater.cc:511] Master 127.0.98.62:43223 requested a full tablet report, sending...
I20260926 20:58:31.963203 14666 ts_manager.cc:194] Registered new tserver with Master: 68eea3b3adcc48c48f75c6a58fb9a287 (127.0.98.3:35901)
I20260926 20:58:31.963730 14666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51091
I20260926 20:58:31.966811 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.958788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53046 (local address 127.0.98.62:43223)
0926 20:58:31.959614 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:31.959617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:31.959626 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:58:31.959656 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:31.959657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:31.959670 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:31.959748 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:31.960542 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.961325 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.962324 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.962614 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.962685 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.964166 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:31.964167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:31.964167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:31.964176 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:31.965692 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:31.966038 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:31.966039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:31.966039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:31.966092 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:31.966340 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:31.966340 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:31.966340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:31.966446 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:31.966559 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:31.966674 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 20:58:31.966742 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":70,"threads_started":1}
I20260926 20:58:31.967597 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:58:31.970398 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.967949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53048 (local address 127.0.98.62:43223)
0926 20:58:31.968000 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:31.968002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:31.968109 (+ 107us) server_negotiation.cc:407] Connection header received
0926 20:58:31.968162 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:31.968162 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:31.968189 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:31.968246 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:58:31.968327 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.968858 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.969621 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.969805 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.969855 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.970036 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:58:31.970152 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:58:31.970215 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:31.970327 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 20:58:31.970346 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 20:58:31.970764 14666 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:58:31.972713 14666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:58:31.975436 14788 tablet_service.cc:1467] Processing CreateTablet for tablet 62774121e872418b956a4e5155507dea (DEFAULT_TABLE table=test-table [id=8430639771b4485d8f6f2c93ef63c93a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:58:31.975620 14788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62774121e872418b956a4e5155507dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:31.975761 14927 tablet_service.cc:1467] Processing CreateTablet for tablet 62774121e872418b956a4e5155507dea (DEFAULT_TABLE table=test-table [id=8430639771b4485d8f6f2c93ef63c93a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:58:31.975979 14927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62774121e872418b956a4e5155507dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:31.977169 14746 tablet_bootstrap.cc:492] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39: Bootstrap starting.
I20260926 20:58:31.977486 14746 tablet_bootstrap.cc:654] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39: Neither blocks nor log segments found. Creating new log.
I20260926 20:58:31.977617 14885 tablet_bootstrap.cc:492] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610: Bootstrap starting.
I20260926 20:58:31.977954 14885 tablet_bootstrap.cc:654] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610: Neither blocks nor log segments found. Creating new log.
I20260926 20:58:31.978473 14746 tablet_bootstrap.cc:492] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39: No bootstrap required, opened a new log
I20260926 20:58:31.978529 14746 ts_tablet_manager.cc:1397] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:58:31.978713 14746 raft_consensus.cc:348] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:31.978807 14746 raft_consensus.cc:374] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:31.978873 14746 raft_consensus.cc:729] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590bf2a9350f4d51a26c85634269ac39, State: Initialized, Role: FOLLOWER
I20260926 20:58:31.978929 14746 consensus_queue.cc:260] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:31.978964 14885 tablet_bootstrap.cc:492] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610: No bootstrap required, opened a new log
I20260926 20:58:31.979007 14885 ts_tablet_manager.cc:1397] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:58:31.979046 14746 ts_tablet_manager.cc:1428] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 20:58:31.979159 14885 raft_consensus.cc:348] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:31.979233 14885 raft_consensus.cc:374] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:31.979292 14885 raft_consensus.cc:729] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7239e877811f4a348557116462308610, State: Initialized, Role: FOLLOWER
I20260926 20:58:31.979354 14885 consensus_queue.cc:260] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:31.979488 14885 ts_tablet_manager.cc:1428] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:31.979820 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.975252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35901 (local address 127.0.0.1:46534)
0926 20:58:31.975310 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:31.975323 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:31.975364 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:31.975767 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:31.975768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:31.975770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:31.975826 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.975828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.977202 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.977203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.978376 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:31.978380 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.979549 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.979550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:31.979623 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.979642 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:58:31.979674 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 20:58:31.979734 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 20:58:31.980225 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:31.975351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46534 (local address 127.0.98.3:35901)
0926 20:58:31.975567 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:31.975573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:31.975585 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:58:31.975629 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:31.975631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:31.975647 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:31.975735 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:31.975916 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.977070 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.978469 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:31.979445 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:31.979997 (+ 552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:31.980091 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:31.980117 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 20:58:31.980155 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":140,"threads_started":1}
I20260926 20:58:31.980962 15063 tablet_service.cc:1467] Processing CreateTablet for tablet 62774121e872418b956a4e5155507dea (DEFAULT_TABLE table=test-table [id=8430639771b4485d8f6f2c93ef63c93a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:58:31.981323 15063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62774121e872418b956a4e5155507dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:31.984172 15138 tablet_bootstrap.cc:492] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287: Bootstrap starting.
I20260926 20:58:31.984922 15138 tablet_bootstrap.cc:654] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287: Neither blocks nor log segments found. Creating new log.
I20260926 20:58:31.985574 15138 log.cc:824] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:31.986423 15138 tablet_bootstrap.cc:492] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287: No bootstrap required, opened a new log
I20260926 20:58:31.986487 15138 ts_tablet_manager.cc:1397] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:58:31.988050 15138 raft_consensus.cc:348] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:31.988212 15138 raft_consensus.cc:374] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:31.988238 15138 raft_consensus.cc:729] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68eea3b3adcc48c48f75c6a58fb9a287, State: Initialized, Role: FOLLOWER
I20260926 20:58:31.988368 15138 consensus_queue.cc:260] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:31.988754 15138 ts_tablet_manager.cc:1428] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:58:31.988875 15129 heartbeater.cc:503] Master 127.0.98.62:43223 was elected leader, sending a full tablet report...
W20260926 20:58:32.062311 14994 tablet.cc:2369] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:58:32.124169 15140 raft_consensus.cc:482] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:58:32.124346 15140 raft_consensus.cc:504] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:32.124845 15140 leader_election.cc:288] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7239e877811f4a348557116462308610 (127.0.98.2:44125), 590bf2a9350f4d51a26c85634269ac39 (127.0.98.1:38519)
I20260926 20:58:32.128149 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:32.125131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51445 (local address 127.0.98.1:38519)
0926 20:58:32.125187 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:32.125189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:32.125200 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:58:32.125247 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:32.125248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:32.125272 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:32.125360 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:32.125540 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:32.126177 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:32.127197 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:32.127639 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:32.127729 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:32.127790 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:32.128016 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 20:58:32.128070 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 20:58:32.128150 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:32.125057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38519 (local address 127.0.98.3:51445)
0926 20:58:32.125112 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:32.125124 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:32.125174 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:32.125364 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:32.125365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:32.125367 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:32.125441 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:32.125443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:32.126271 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:32.126272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:32.127082 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:32.127084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:32.127827 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:32.127828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:32.127923 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:32.127927 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:58:32.127983 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:58:32.128035 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260926 20:58:32.128224 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:32.125153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56529 (local address 127.0.98.2:44125)
0926 20:58:32.125206 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:32.125208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:32.125476 (+ 268us) server_negotiation.cc:407] Connection header received
0926 20:58:32.125507 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:32.125508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:32.125534 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:32.125596 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:32.125764 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:32.126283 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:32.127273 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:32.127830 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:32.127904 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:32.127986 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:32.128101 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 20:58:32.128151 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:58:32.128496 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:32.125066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44125 (local address 127.0.98.3:56529)
0926 20:58:32.125370 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:32.125384 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:32.125440 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:32.125618 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:32.125618 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:32.125620 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:32.125684 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:32.125686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:32.126384 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:32.126385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:32.127113 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:32.127115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:32.127922 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:32.127923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:32.128012 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:32.128016 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:58:32.128076 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 20:58:32.128116 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":109,"thread_start_us":141,"threads_started":1}
I20260926 20:58:32.128594 14808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62774121e872418b956a4e5155507dea" candidate_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "590bf2a9350f4d51a26c85634269ac39" is_pre_election: true
I20260926 20:58:32.128805 14808 raft_consensus.cc:2454] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68eea3b3adcc48c48f75c6a58fb9a287 in term 0.
I20260926 20:58:32.128868 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62774121e872418b956a4e5155507dea" candidate_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7239e877811f4a348557116462308610" is_pre_election: true
I20260926 20:58:32.129011 14947 raft_consensus.cc:2454] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68eea3b3adcc48c48f75c6a58fb9a287 in term 0.
I20260926 20:58:32.129092 15017 leader_election.cc:302] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 590bf2a9350f4d51a26c85634269ac39, 68eea3b3adcc48c48f75c6a58fb9a287; no voters:
I20260926 20:58:32.129254 15140 raft_consensus.cc:2781] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:58:32.129323 15140 raft_consensus.cc:482] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:58:32.129341 15140 raft_consensus.cc:3037] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:32.130111 15140 raft_consensus.cc:504] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:32.130271 15140 leader_election.cc:288] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [CANDIDATE]: Term 1 election: Requested vote from peers 7239e877811f4a348557116462308610 (127.0.98.2:44125), 590bf2a9350f4d51a26c85634269ac39 (127.0.98.1:38519)
I20260926 20:58:32.130424 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62774121e872418b956a4e5155507dea" candidate_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7239e877811f4a348557116462308610"
I20260926 20:58:32.130471 14808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62774121e872418b956a4e5155507dea" candidate_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "590bf2a9350f4d51a26c85634269ac39"
I20260926 20:58:32.130517 14947 raft_consensus.cc:3037] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:32.130599 14808 raft_consensus.cc:3037] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:32.131278 14947 raft_consensus.cc:2454] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68eea3b3adcc48c48f75c6a58fb9a287 in term 1.
I20260926 20:58:32.131407 14808 raft_consensus.cc:2454] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68eea3b3adcc48c48f75c6a58fb9a287 in term 1.
I20260926 20:58:32.131464 15016 leader_election.cc:302] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68eea3b3adcc48c48f75c6a58fb9a287, 7239e877811f4a348557116462308610; no voters:
I20260926 20:58:32.131554 15140 raft_consensus.cc:2781] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:58:32.131729 15140 raft_consensus.cc:686] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [term 1 LEADER]: Becoming Leader. State: Replica: 68eea3b3adcc48c48f75c6a58fb9a287, State: Running, Role: LEADER
I20260926 20:58:32.131812 15140 consensus_queue.cc:237] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:32.132850 14665 catalog_manager.cc:5495] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68eea3b3adcc48c48f75c6a58fb9a287 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } health_report { overall_health: UNKNOWN } } }
I20260926 20:58:32.164320 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:32.161848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46536 (local address 127.0.98.3:35901)
0926 20:58:32.161920 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:32.161922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:32.161931 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:58:32.161974 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:32.161975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:32.161995 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:32.162075 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:58:32.162237 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:32.162727 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:32.163505 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:32.163707 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:32.163784 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:32.163972 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:58:32.164078 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:58:32.164171 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:32.164195 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 20:58:32.164219 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:58:32.166504 14947 raft_consensus.cc:1261] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610 [term 1 FOLLOWER]: Refusing update from remote peer 68eea3b3adcc48c48f75c6a58fb9a287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:58:32.166504 14808 raft_consensus.cc:1261] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Refusing update from remote peer 68eea3b3adcc48c48f75c6a58fb9a287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:58:32.166867 15140 consensus_queue.cc:1035] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:58:32.167006 15142 consensus_queue.cc:1035] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:58:32.168927 14855 tablet.cc:2369] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:58:32.169413 15146 mvcc.cc:205] Tried to move back new op lower bound from 7333709054630989824 to 7333709054492798976. Current Snapshot: MvccSnapshot[applied={T|T < 7333709054630989824 or (T in {7333709054630989824})}]
I20260926 20:58:32.169831 15147 mvcc.cc:205] Tried to move back new op lower bound from 7333709054630989824 to 7333709054492798976. Current Snapshot: MvccSnapshot[applied={T|T < 7333709054630989824}]
I20260926 20:58:32.171496 14666 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:58:32.171604 14666 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:58:32.172307 14666 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e: Sending DeleteTablet for 3 replicas of tablet 62774121e872418b956a4e5155507dea
I20260926 20:58:32.172645 14788 tablet_service.cc:1514] Processing DeleteTablet for tablet 62774121e872418b956a4e5155507dea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:58:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34504
I20260926 20:58:32.172647 15063 tablet_service.cc:1514] Processing DeleteTablet for tablet 62774121e872418b956a4e5155507dea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:58:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46534
I20260926 20:58:32.172765 14863 tablet_replica.cc:330] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39: stopping tablet replica
I20260926 20:58:32.172819 14863 raft_consensus.cc:2229] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:58:32.172870 14863 raft_consensus.cc:2258] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:58:32.173018 15154 tablet_replica.cc:330] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287: stopping tablet replica
I20260926 20:58:32.173070 14927 tablet_service.cc:1514] Processing DeleteTablet for tablet 62774121e872418b956a4e5155507dea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:58:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56902
I20260926 20:58:32.173112 15154 raft_consensus.cc:2229] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:58:32.173146 15000 tablet_replica.cc:330] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610: stopping tablet replica
I20260926 20:58:32.173182 15000 raft_consensus.cc:2229] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:58:32.173166 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 14704
I20260926 20:58:32.173233 15000 raft_consensus.cc:2258] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:58:32.173640 15154 raft_consensus.cc:2258] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:58:32.177452 15000 ts_tablet_manager.cc:1905] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:58:32.177546 15154 ts_tablet_manager.cc:1905] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:58:32.178506 14651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38519 (error 108)
I20260926 20:58:32.179563 15000 ts_tablet_manager.cc:1918] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:58:32.179564 15154 ts_tablet_manager.cc:1918] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:58:32.179631 15000 log.cc:1196] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/62774121e872418b956a4e5155507dea
I20260926 20:58:32.179661 15154 log.cc:1196] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/62774121e872418b956a4e5155507dea
I20260926 20:58:32.179864 15000 ts_tablet_manager.cc:1939] T 62774121e872418b956a4e5155507dea P 7239e877811f4a348557116462308610: Deleting consensus metadata
I20260926 20:58:32.179982 15154 ts_tablet_manager.cc:1939] T 62774121e872418b956a4e5155507dea P 68eea3b3adcc48c48f75c6a58fb9a287: Deleting consensus metadata
I20260926 20:58:32.179989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 14859
I20260926 20:58:32.181957 14652 catalog_manager.cc:4841] TS 68eea3b3adcc48c48f75c6a58fb9a287 (127.0.98.3:35901): tablet 62774121e872418b956a4e5155507dea (table test-table [id=8430639771b4485d8f6f2c93ef63c93a]) successfully deleted
I20260926 20:58:32.184944 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:32.184785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38519 (local address 127.0.0.1:34518)
0926 20:58:32.184851 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:32.184901 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 20:58:32.185242 14651 catalog_manager.cc:4584] TS 590bf2a9350f4d51a26c85634269ac39 (127.0.98.1:38519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62774121e872418b956a4e5155507dea: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38519: connect: Connection refused (error 111)
I20260926 20:58:32.186422 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 14996
I20260926 20:58:32.186539 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:32.186414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44125 (local address 127.0.0.1:56918)
0926 20:58:32.186471 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:32.186500 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 20:58:32.191552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 14633
I20260926 20:58:32.197217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43223
--webserver_interface=127.0.98.62
--webserver_port=34973
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:58:32.281534 15156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:32.281770 15156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:32.281796 15156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:32.283069 15156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:58:32.283126 15156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:32.283141 15156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:32.283154 15156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:58:32.283166 15156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:58:32.284755 15156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:32.284983 15156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:32.285171 15156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:32.290107 15164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:32.290132 15161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:32.290287 15156 server_base.cc:1034] running on GCE node
W20260926 20:58:32.290361 15162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:32.290594 15156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:32.290915 15156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:32.292138 15156 hybrid_clock.cc:648] HybridClock initialized: now 1790456312292121 us; error 46 us; skew 500 ppm
Sep 26 20:58:32 dist-test-slave-fl7g krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:32.295225 15156 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:58:32.295742 15156 webserver.cc:460] Webserver started at http://127.0.98.62:34973/ using document root <none> and password file <none>
I20260926 20:58:32.295987 15156 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:32.296060 15156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:32.297609 15156 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:32.298329 15171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:32.298527 15156 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:32.298643 15156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "2ecaaac71a38497bb046a4a8234e153e"
format_stamp: "Formatted at 2026-09-26 20:58:30 on dist-test-slave-fl7g"
server_key: "6f1c0a7e937ab89e95da753e77be4b4b"
server_key_iv: "bdf38ca53ef7f04caad9595c1b08c5f0"
server_key_version: "encryptionkey@0"
I20260926 20:58:32.299021 15156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:32.320904 15156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:32.323254 15156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:32.323436 15156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:32.327400 15156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43223
I20260926 20:58:32.327504 15223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43223 every 8 connection(s)
I20260926 20:58:32.327898 15156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:58:32.329903 15224 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:58:32.330518 15224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e: Bootstrap starting.
I20260926 20:58:32.332912 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 15156
I20260926 20:58:32.333202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38519
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43223
--webserver_port=38861
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:58:32.339218 15224 log.cc:824] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:32.341710 15224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:58:32.342099 15224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e: Bootstrap complete.
I20260926 20:58:32.344797 15224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecaaac71a38497bb046a4a8234e153e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43223 } }
I20260926 20:58:32.345088 15224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecaaac71a38497bb046a4a8234e153e, State: Initialized, Role: FOLLOWER
I20260926 20:58:32.345271 15224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecaaac71a38497bb046a4a8234e153e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43223 } }
I20260926 20:58:32.345345 15224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:58:32.345398 15224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:58:32.345449 15224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:58:32.346619 15224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecaaac71a38497bb046a4a8234e153e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43223 } }
I20260926 20:58:32.346748 15224 leader_election.cc:302] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ecaaac71a38497bb046a4a8234e153e; no voters:
I20260926 20:58:32.346961 15224 leader_election.cc:288] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 20:58:32.347003 15228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 3 FOLLOWER]: Leader election won for term 3
I20260926 20:58:32.347142 15228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [term 3 LEADER]: Becoming Leader. State: Replica: 2ecaaac71a38497bb046a4a8234e153e, State: Running, Role: LEADER
I20260926 20:58:32.347244 15228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecaaac71a38497bb046a4a8234e153e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43223 } }
I20260926 20:58:32.347277 15224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:58:32.347553 15228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ecaaac71a38497bb046a4a8234e153e. Latest consensus state: current_term: 3 leader_uuid: "2ecaaac71a38497bb046a4a8234e153e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecaaac71a38497bb046a4a8234e153e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43223 } } }
I20260926 20:58:32.347625 15228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [sys.catalog]: This master's current role is: LEADER
I20260926 20:58:32.347786 15229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2ecaaac71a38497bb046a4a8234e153e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecaaac71a38497bb046a4a8234e153e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43223 } } }
I20260926 20:58:32.347865 15229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ecaaac71a38497bb046a4a8234e153e [sys.catalog]: This master's current role is: LEADER
I20260926 20:58:32.348363 15233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:58:32.349275 15233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:58:32.349382 15233 catalog_manager.cc:1259] Loaded cluster ID: 96932c0687ab4665bc2f4647e988aea0
I20260926 20:58:32.349416 15233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 20:58:32.349648 15233 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa45ec9f9ad google::LogMessage::Fail() at ??:0
@ 0x7fa45eca288a google::LogMessage::SendToLog() at ??:0
@ 0x7fa45ec9f4de google::LogMessage::Flush() at ??:0
@ 0x7fa45eca0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa466bc00f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fa466bc98b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa460b6d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa460b62954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa4619c66db start_thread at ??:0
@ 0x7fa45d79d61f clone at ??:0
W20260926 20:58:32.434101 15226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:32.434322 15226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:32.434350 15226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:32.435760 15226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:32.435833 15226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:32.435878 15226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:58:32.437834 15226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38861
--tserver_master_addrs=127.0.98.62:43223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:32.438150 15226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:32.438412 15226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:32.443817 15260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:32.443817 15263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:32.443811 15261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:32.444401 15226 server_base.cc:1034] running on GCE node
I20260926 20:58:32.444667 15226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:32.444944 15226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:32.446159 15226 hybrid_clock.cc:648] HybridClock initialized: now 1790456312446139 us; error 52 us; skew 500 ppm
Sep 26 20:58:32 dist-test-slave-fl7g krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:32.448876 15226 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:58:32.449358 15226 webserver.cc:460] Webserver started at http://127.0.98.1:38861/ using document root <none> and password file <none>
I20260926 20:58:32.449625 15226 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:32.449697 15226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:32.451190 15226 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:32.452014 15270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:32.452276 15226 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:32.452395 15226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "590bf2a9350f4d51a26c85634269ac39"
format_stamp: "Formatted at 2026-09-26 20:58:30 on dist-test-slave-fl7g"
server_key: "958040b889bbb419296dd5840dba9fe5"
server_key_iv: "874dad09f9aaa11f91c50e6aea07c452"
server_key_version: "encryptionkey@0"
I20260926 20:58:32.453076 15226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:32.469856 15226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:32.472668 15226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:32.472852 15226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:32.473120 15226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:58:32.473599 15277 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:58:32.475869 15226 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:58:32.475978 15226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260926 20:58:32.476042 15226 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:58:32.476843 15226 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:58:32.476907 15226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260926 20:58:32.476991 15277 tablet_bootstrap.cc:492] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39: Bootstrap starting.
I20260926 20:58:32.484524 15226 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38519
I20260926 20:58:32.484704 15384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38519 every 8 connection(s)
I20260926 20:58:32.484985 15226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:58:32.488447 15277 log.cc:824] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:32.488778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 15226
I20260926 20:58:32.489051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44125
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43223
--webserver_port=41755
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:58:32.491343 15277 tablet_bootstrap.cc:492] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:58:32.491693 15277 tablet_bootstrap.cc:492] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39: Bootstrap complete.
I20260926 20:58:32.493002 15277 ts_tablet_manager.cc:1397] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.008s
I20260926 20:58:32.494469 15277 raft_consensus.cc:348] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:32.494831 15277 raft_consensus.cc:729] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590bf2a9350f4d51a26c85634269ac39, State: Initialized, Role: FOLLOWER
I20260926 20:58:32.495020 15277 consensus_queue.cc:260] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:32.495349 15277 ts_tablet_manager.cc:1428] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260926 20:58:32.574278 15389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:32.574524 15389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:32.574566 15389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:32.576540 15389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:32.576623 15389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:32.576674 15389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:58:32.578329 15389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41755
--tserver_master_addrs=127.0.98.62:43223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:32.578649 15389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:32.578914 15389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:32.579865 15387 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:58:32.485729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:54609)
0926 20:58:32.485921 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:32.485967 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:32.486457 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:32.579654 (+ 93197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":112,"threads_started":1}
W20260926 20:58:32.580199 15385 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43223 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43223: Client connection negotiation failed: client connection to 127.0.98.62:43223: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:58:32.580647 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:32.580477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:54811)
0926 20:58:32.580551 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:32.580610 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 20:58:32.581048 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:32.580933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:49519)
0926 20:58:32.580987 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:32.581024 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 20:58:32.581213 15385 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:58:32.584765 15395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:32.584789 15396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:32.584765 15398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:32.585023 15389 server_base.cc:1034] running on GCE node
I20260926 20:58:32.585235 15389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:32.585453 15389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:32.586640 15389 hybrid_clock.cc:648] HybridClock initialized: now 1790456312586636 us; error 49 us; skew 500 ppm
Sep 26 20:58:32 dist-test-slave-fl7g krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:32.589499 15389 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:58:32.590010 15389 webserver.cc:460] Webserver started at http://127.0.98.2:41755/ using document root <none> and password file <none>
I20260926 20:58:32.590278 15389 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:32.590356 15389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:32.591842 15389 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:58:32.592574 15405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:32.592796 15389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:58:32.592892 15389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "7239e877811f4a348557116462308610"
format_stamp: "Formatted at 2026-09-26 20:58:31 on dist-test-slave-fl7g"
server_key: "4c2fbb8575f918443b6c8f406e3cde4d"
server_key_iv: "67d5ec67a0e70b74aa685783d9bf3a5a"
server_key_version: "encryptionkey@0"
I20260926 20:58:32.593284 15389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:32.606448 15389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:32.608572 15389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:32.608753 15389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:32.609045 15389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:58:32.609381 15389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:58:32.609439 15389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:32.609483 15389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:58:32.609509 15389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:32.616041 15389 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44125
I20260926 20:58:32.616083 15518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44125 every 8 connection(s)
I20260926 20:58:32.616575 15389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:58:32.617787 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:32.617364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:50109)
0926 20:58:32.617527 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:32.617604 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
W20260926 20:58:32.618232 15519 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43223 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43223: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
I20260926 20:58:32.618604 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:32.618494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:59305)
0926 20:58:32.618551 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:32.618579 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 20:58:32.618942 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:32.618854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:57615)
0926 20:58:32.618906 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:32.618928 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 20:58:32.619086 15519 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:58:32.624743 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 15389
I20260926 20:58:32.625061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35901
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43223
--webserver_port=34089
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:58:32.709960 15522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:32.710215 15522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:32.710243 15522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:32.711570 15522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:32.711630 15522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:32.711661 15522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:58:32.713392 15522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34089
--tserver_master_addrs=127.0.98.62:43223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:32.713682 15522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:32.713955 15522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:32.719456 15528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:32.719635 15522 server_base.cc:1034] running on GCE node
W20260926 20:58:32.719461 15530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:32.719434 15527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:32.719955 15522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:32.720230 15522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:32.721369 15522 hybrid_clock.cc:648] HybridClock initialized: now 1790456312721361 us; error 37 us; skew 500 ppm
Sep 26 20:58:32 dist-test-slave-fl7g krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:32.724036 15522 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:58:32.724588 15522 webserver.cc:460] Webserver started at http://127.0.98.3:34089/ using document root <none> and password file <none>
I20260926 20:58:32.724831 15522 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:32.724881 15522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:32.726305 15522 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:58:32.727017 15537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:32.727200 15522 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:58:32.727317 15522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "68eea3b3adcc48c48f75c6a58fb9a287"
format_stamp: "Formatted at 2026-09-26 20:58:31 on dist-test-slave-fl7g"
server_key: "69880703044e878e0c0cad46a641aa55"
server_key_iv: "c892406223c96147cb40737ee70fdf17"
server_key_version: "encryptionkey@0"
I20260926 20:58:32.727691 15522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:32.744760 15522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:32.746896 15522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:32.747012 15522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:32.747186 15522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:58:32.747532 15522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:58:32.747570 15522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:32.747618 15522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:58:32.747645 15522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:32.753602 15522 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35901
I20260926 20:58:32.753680 15650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35901 every 8 connection(s)
I20260926 20:58:32.754058 15522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:58:32.755200 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:32.754799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:43353)
0926 20:58:32.754957 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:32.755032 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
W20260926 20:58:32.755578 15651 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43223 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43223: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
I20260926 20:58:32.755870 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:32.755771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:36695)
0926 20:58:32.755822 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:32.755850 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 20:58:32.756206 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:32.756114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:47207)
0926 20:58:32.756174 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:32.756193 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 20:58:32.756398 15651 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:58:32.760746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 15522
I20260926 20:58:33.581976 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:33.581693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:59011)
0926 20:58:33.581862 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:33.581914 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":118,"threads_started":1}
I20260926 20:58:33.619830 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:33.619472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:60077)
0926 20:58:33.619651 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:33.619698 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260926 20:58:33.757002 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:33.756791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:47531)
0926 20:58:33.756909 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:33.756949 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260926 20:58:34.119488 15658 raft_consensus.cc:482] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:58:34.119607 15658 raft_consensus.cc:504] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:34.119987 15658 leader_election.cc:288] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68eea3b3adcc48c48f75c6a58fb9a287 (127.0.98.3:35901), 7239e877811f4a348557116462308610 (127.0.98.2:44125)
Sep 26 20:58:34 dist-test-slave-fl7g krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 26 20:58:34 dist-test-slave-fl7g krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 20:58:34.134913 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:34.120463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47383 (local address 127.0.98.2:44125)
0926 20:58:34.120615 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:34.120623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:34.120640 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:58:34.121039 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:34.121041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:34.121083 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:34.121266 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:34.122006 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:34.122863 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:34.123896 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:34.124323 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:34.124472 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:34.131105 (+ 6633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:34.131161 (+ 56us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:34.131162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:34.131175 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:34.133036 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:34.133560 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:34.133565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:34.133565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:34.133677 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:34.133961 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:34.133962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:34.133962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:34.134234 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:34.134348 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:34.134653 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 20:58:34.134789 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260926 20:58:34.134922 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:34.120280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48207 (local address 127.0.98.3:35901)
0926 20:58:34.120461 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:34.120473 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:34.120527 (+ 54us) server_negotiation.cc:407] Connection header received
0926 20:58:34.120994 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:34.120996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:34.121039 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:34.121157 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:34.122036 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:34.122878 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:34.123916 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:34.124323 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:34.124491 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:34.130803 (+ 6312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:34.130810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:34.130811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:34.130827 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:34.132962 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:34.133560 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:34.133564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:34.133565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:34.133671 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:34.133960 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:34.133961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:34.133961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:34.134235 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:34.134324 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:34.134641 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 20:58:34.134779 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":46,"threads_started":1}
I20260926 20:58:34.134900 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:34.120129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35901 (local address 127.0.98.1:48207)
0926 20:58:34.120297 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:34.120311 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:34.120370 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:34.121237 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:34.121240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:34.121665 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:34.121874 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:34.121881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:34.123008 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:34.123009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:34.123799 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:34.123802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:34.124456 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:34.124457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:34.124574 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:34.128813 (+ 4239us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:34.128825 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:34.130627 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:34.133071 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:34.133084 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:34.133087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:34.133459 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:34.133744 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:34.133745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:34.133746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:34.133887 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:34.134317 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:34.134325 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:34.134467 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 20:58:34.134583 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 20:58:34.134802 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":74,"thread_start_us":57,"threads_started":1}
I20260926 20:58:34.135056 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:34.120275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44125 (local address 127.0.98.1:47383)
0926 20:58:34.120394 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:34.120425 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:34.120465 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:34.121245 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:34.121246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:34.121667 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:34.121874 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:34.121881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:34.122970 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:34.122971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:34.123799 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:34.123802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:34.124451 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:34.124452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:34.124574 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:34.128814 (+ 4240us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:34.128824 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:34.130954 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:34.133146 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:34.133147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:34.133147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:34.133459 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:34.133771 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:34.133772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:34.133772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:34.133887 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:34.134345 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:34.134346 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:34.134465 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 20:58:34.134582 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:58:34.134893 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":156,"thread_start_us":72,"threads_started":1}
W20260926 20:58:34.136561 15271 leader_election.cc:341] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7239e877811f4a348557116462308610 (127.0.98.2:44125): Not found: Tablet not found: 62774121e872418b956a4e5155507dea
W20260926 20:58:34.136653 15273 leader_election.cc:341] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68eea3b3adcc48c48f75c6a58fb9a287 (127.0.98.3:35901): Not found: Tablet not found: 62774121e872418b956a4e5155507dea
I20260926 20:58:34.136757 15273 leader_election.cc:302] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 590bf2a9350f4d51a26c85634269ac39; no voters: 68eea3b3adcc48c48f75c6a58fb9a287, 7239e877811f4a348557116462308610
I20260926 20:58:34.136897 15658 raft_consensus.cc:2726] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:58:34.582863 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:34.582661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:54471)
0926 20:58:34.582769 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:34.582821 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 20:58:34.620879 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:34.620604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:42743)
0926 20:58:34.620759 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:34.620824 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260926 20:58:34.757807 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:34.757532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:36175)
0926 20:58:34.757722 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:34.757763 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":137,"threads_started":1}
I20260926 20:58:35.583878 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:35.583567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:46197)
0926 20:58:35.583786 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:35.583832 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":99,"threads_started":1}
I20260926 20:58:35.621620 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:35.621395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:35209)
0926 20:58:35.621530 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:35.621569 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
I20260926 20:58:35.758641 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:35.758393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:53439)
0926 20:58:35.758543 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:35.758592 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20260926 20:58:36.039546 15668 raft_consensus.cc:482] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:58:36.039700 15668 raft_consensus.cc:504] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:36.039948 15668 leader_election.cc:288] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68eea3b3adcc48c48f75c6a58fb9a287 (127.0.98.3:35901), 7239e877811f4a348557116462308610 (127.0.98.2:44125)
W20260926 20:58:36.040467 15273 leader_election.cc:341] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68eea3b3adcc48c48f75c6a58fb9a287 (127.0.98.3:35901): Not found: Tablet not found: 62774121e872418b956a4e5155507dea
W20260926 20:58:36.040547 15271 leader_election.cc:341] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7239e877811f4a348557116462308610 (127.0.98.2:44125): Not found: Tablet not found: 62774121e872418b956a4e5155507dea
I20260926 20:58:36.040598 15271 leader_election.cc:302] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 590bf2a9350f4d51a26c85634269ac39; no voters: 68eea3b3adcc48c48f75c6a58fb9a287, 7239e877811f4a348557116462308610
I20260926 20:58:36.040668 15668 raft_consensus.cc:2726] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:58:36.584875 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:36.584625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:52253)
0926 20:58:36.584779 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:36.584827 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260926 20:58:36.622525 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:36.622268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:53037)
0926 20:58:36.622420 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:36.622472 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20260926 20:58:36.759523 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:36.759260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:50869)
0926 20:58:36.759423 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:36.759476 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":95,"threads_started":1}
I20260926 20:58:37.585860 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:37.585631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:40295)
0926 20:58:37.585769 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:37.585813 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260926 20:58:37.623476 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:37.623209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:33223)
0926 20:58:37.623371 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:37.623424 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":69,"threads_started":1}
I20260926 20:58:37.760336 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:37.760109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:34673)
0926 20:58:37.760242 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:37.760289 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260926 20:58:38.106768 15675 raft_consensus.cc:482] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:58:38.106869 15675 raft_consensus.cc:504] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:38.107069 15675 leader_election.cc:288] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68eea3b3adcc48c48f75c6a58fb9a287 (127.0.98.3:35901), 7239e877811f4a348557116462308610 (127.0.98.2:44125)
W20260926 20:58:38.107508 15273 leader_election.cc:341] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68eea3b3adcc48c48f75c6a58fb9a287 (127.0.98.3:35901): Not found: Tablet not found: 62774121e872418b956a4e5155507dea
W20260926 20:58:38.107601 15271 leader_election.cc:341] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7239e877811f4a348557116462308610 (127.0.98.2:44125): Not found: Tablet not found: 62774121e872418b956a4e5155507dea
I20260926 20:58:38.107653 15271 leader_election.cc:302] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 590bf2a9350f4d51a26c85634269ac39; no voters: 68eea3b3adcc48c48f75c6a58fb9a287, 7239e877811f4a348557116462308610
I20260926 20:58:38.107743 15675 raft_consensus.cc:2726] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:58:38.586772 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:38.586481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:40663)
0926 20:58:38.586674 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:38.586723 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260926 20:58:38.624254 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:38.624025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:57987)
0926 20:58:38.624170 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:38.624207 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20260926 20:58:38.761138 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:38.760881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:45803)
0926 20:58:38.761028 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:38.761085 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260926 20:58:39.587663 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:39.587377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:40489)
0926 20:58:39.587542 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:39.587589 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":66,"threads_started":1}
I20260926 20:58:39.624992 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:39.624733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:44451)
0926 20:58:39.624903 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:39.624943 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":63,"threads_started":1}
I20260926 20:58:39.761832 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:39.761577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:38385)
0926 20:58:39.761741 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:39.761787 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260926 20:58:40.588569 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:40.588298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:40127)
0926 20:58:40.588474 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:40.588523 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":72,"threads_started":1}
I20260926 20:58:40.625765 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:40.625519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:55481)
0926 20:58:40.625675 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:40.625718 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260926 20:58:40.762486 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:40.762280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:34983)
0926 20:58:40.762404 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:40.762440 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260926 20:58:41.009018 15685 raft_consensus.cc:482] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:58:41.009116 15685 raft_consensus.cc:504] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:41.009338 15685 leader_election.cc:288] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68eea3b3adcc48c48f75c6a58fb9a287 (127.0.98.3:35901), 7239e877811f4a348557116462308610 (127.0.98.2:44125)
W20260926 20:58:41.009719 15273 leader_election.cc:341] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68eea3b3adcc48c48f75c6a58fb9a287 (127.0.98.3:35901): Not found: Tablet not found: 62774121e872418b956a4e5155507dea
W20260926 20:58:41.009817 15271 leader_election.cc:341] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7239e877811f4a348557116462308610 (127.0.98.2:44125): Not found: Tablet not found: 62774121e872418b956a4e5155507dea
I20260926 20:58:41.009912 15271 leader_election.cc:302] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 590bf2a9350f4d51a26c85634269ac39; no voters: 68eea3b3adcc48c48f75c6a58fb9a287, 7239e877811f4a348557116462308610
I20260926 20:58:41.009999 15685 raft_consensus.cc:2726] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:58:41.589425 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:41.589159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:48217)
0926 20:58:41.589330 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:41.589377 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
I20260926 20:58:41.626499 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:41.626276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:54907)
0926 20:58:41.626402 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:41.626448 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260926 20:58:41.763245 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:41.763027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:59379)
0926 20:58:41.763160 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:41.763199 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260926 20:58:42.590376 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:42.590041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:43647)
0926 20:58:42.590230 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:42.590314 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260926 20:58:42.627310 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:42.627065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:58487)
0926 20:58:42.627208 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:42.627259 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
I20260926 20:58:42.764038 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:42.763786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:57303)
0926 20:58:42.763943 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:42.763985 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260926 20:58:43.591262 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:43.591003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:42389)
0926 20:58:43.591164 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:43.591213 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
I20260926 20:58:43.628171 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:43.627928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:33039)
0926 20:58:43.628060 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:43.628107 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260926 20:58:43.764816 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:43.764565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:53845)
0926 20:58:43.764714 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:43.764770 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260926 20:58:44.592203 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:44.591942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:55523)
0926 20:58:44.592105 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:44.592153 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":78,"threads_started":1}
I20260926 20:58:44.628962 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:44.628715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:32839)
0926 20:58:44.628857 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:44.628915 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260926 20:58:44.765645 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:44.765381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:52127)
0926 20:58:44.765536 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:44.765593 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260926 20:58:45.593164 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:45.592830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:36601)
0926 20:58:45.593013 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:45.593107 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":88,"threads_started":1}
I20260926 20:58:45.629834 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:45.629559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:36411)
0926 20:58:45.629738 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:45.629785 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
I20260926 20:58:45.766407 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:45.766192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:58979)
0926 20:58:45.766314 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:45.766353 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260926 20:58:46.593935 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:46.593712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:52927)
0926 20:58:46.593844 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:46.593886 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260926 20:58:46.630753 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:46.630412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:45587)
0926 20:58:46.630607 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:46.630689 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":83,"threads_started":1}
I20260926 20:58:46.767277 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:46.766984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:35181)
0926 20:58:46.767175 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:46.767227 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
I20260926 20:58:46.813864 15704 raft_consensus.cc:482] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:58:46.813975 15704 raft_consensus.cc:504] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68eea3b3adcc48c48f75c6a58fb9a287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35901 } } peers { permanent_uuid: "7239e877811f4a348557116462308610" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44125 } } peers { permanent_uuid: "590bf2a9350f4d51a26c85634269ac39" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38519 } }
I20260926 20:58:46.814162 15704 leader_election.cc:288] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68eea3b3adcc48c48f75c6a58fb9a287 (127.0.98.3:35901), 7239e877811f4a348557116462308610 (127.0.98.2:44125)
W20260926 20:58:46.814654 15273 leader_election.cc:341] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68eea3b3adcc48c48f75c6a58fb9a287 (127.0.98.3:35901): Not found: Tablet not found: 62774121e872418b956a4e5155507dea
W20260926 20:58:46.814750 15271 leader_election.cc:341] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7239e877811f4a348557116462308610 (127.0.98.2:44125): Not found: Tablet not found: 62774121e872418b956a4e5155507dea
I20260926 20:58:46.814836 15271 leader_election.cc:302] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 590bf2a9350f4d51a26c85634269ac39; no voters: 68eea3b3adcc48c48f75c6a58fb9a287, 7239e877811f4a348557116462308610
I20260926 20:58:46.814942 15704 raft_consensus.cc:2726] T 62774121e872418b956a4e5155507dea P 590bf2a9350f4d51a26c85634269ac39 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:58:47.594687 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:47.594445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.1:52707)
0926 20:58:47.594594 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:47.594642 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
I20260926 20:58:47.631578 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:47.631324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.2:42295)
0926 20:58:47.631472 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:47.631521 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":59,"threads_started":1}
I20260926 20:58:47.762241 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 15226
I20260926 20:58:47.767136 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 15389
I20260926 20:58:47.768864 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:47.768634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43223 (local address 127.0.98.3:32791)
0926 20:58:47.768780 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:47.768822 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260926 20:58:47.771589 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 15522
2026-09-26T20:58:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18435 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:58:47 dist-test-slave-fl7g krb5kdc[15710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:58:47 dist-test-slave-fl7g krb5kdc[15710](info): set up 2 sockets
Sep 26 20:58:47 dist-test-slave-fl7g krb5kdc[15710](info): commencing operation
krb5kdc: starting...
W20260926 20:58:48.816828 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:58:48 dist-test-slave-fl7g krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:58:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:58:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:58:48.944203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35223
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:58:49.027814 15723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:49.028064 15723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:49.028090 15723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:49.029462 15723 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:58:49.029510 15723 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:58:49.029523 15723 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:58:49.029536 15723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:58:49.029549 15723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:49.029562 15723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:49.029574 15723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:58:49.029634 15723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:58:49.031222 15723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:49.031523 15723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:49.031782 15723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:58:49.037143 15723 server_base.cc:1034] running on GCE node
W20260926 20:58:49.037098 15731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:49.037104 15729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:49.037127 15728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:49.037714 15723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:49.038043 15723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:49.039227 15723 hybrid_clock.cc:648] HybridClock initialized: now 1790456329039214 us; error 33 us; skew 500 ppm
Sep 26 20:58:49 dist-test-slave-fl7g krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:49.041859 15723 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:58:49.042268 15723 webserver.cc:460] Webserver started at http://127.0.98.62:41187/ using document root <none> and password file <none>
I20260926 20:58:49.042459 15723 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:49.042498 15723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:49.042598 15723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:58:49.043635 15723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "a41d46a69bda4043a0b83db9c5a2fda5"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "921261bbb787c3fa6c9527d7f1498290"
server_key_iv: "907bec97c4f824e8a6d6bfac27f166eb"
server_key_version: "encryptionkey@0"
I20260926 20:58:49.043944 15723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "a41d46a69bda4043a0b83db9c5a2fda5"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "921261bbb787c3fa6c9527d7f1498290"
server_key_iv: "907bec97c4f824e8a6d6bfac27f166eb"
server_key_version: "encryptionkey@0"
I20260926 20:58:49.045399 15723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:58:49.046336 15738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:49.046566 15723 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:58:49.046674 15723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "a41d46a69bda4043a0b83db9c5a2fda5"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "921261bbb787c3fa6c9527d7f1498290"
server_key_iv: "907bec97c4f824e8a6d6bfac27f166eb"
server_key_version: "encryptionkey@0"
I20260926 20:58:49.046744 15723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:49.068043 15723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:49.070649 15723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:49.070770 15723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:49.074857 15723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35223
I20260926 20:58:49.074952 15790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35223 every 8 connection(s)
I20260926 20:58:49.075318 15723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:58:49.076352 15791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:49.078856 15791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: Bootstrap starting.
I20260926 20:58:49.079520 15791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: Neither blocks nor log segments found. Creating new log.
I20260926 20:58:49.079746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 15723
I20260926 20:58:49.079838 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:58:49.079862 15791 log.cc:824] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:49.080060 392 external_mini_cluster.cc:1428] Setting key b8384b919dade9d046bf0dfddb63a8ba
I20260926 20:58:49.080736 15791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: No bootstrap required, opened a new log
I20260926 20:58:49.083442 15791 raft_consensus.cc:348] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260926 20:58:49.083630 15791 raft_consensus.cc:374] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:49.083670 15791 raft_consensus.cc:729] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41d46a69bda4043a0b83db9c5a2fda5, State: Initialized, Role: FOLLOWER
I20260926 20:58:49.083796 15791 consensus_queue.cc:260] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260926 20:58:49.083874 15791 raft_consensus.cc:388] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:58:49.083904 15791 raft_consensus.cc:482] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:58:49.083947 15791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:49.084714 15791 raft_consensus.cc:504] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260926 20:58:49.084808 15791 leader_election.cc:302] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a41d46a69bda4043a0b83db9c5a2fda5; no voters:
I20260926 20:58:49.085003 15791 leader_election.cc:288] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:58:49.085057 15796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:58:49.085273 15796 raft_consensus.cc:686] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 1 LEADER]: Becoming Leader. State: Replica: a41d46a69bda4043a0b83db9c5a2fda5, State: Running, Role: LEADER
I20260926 20:58:49.085383 15791 sys_catalog.cc:564] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:58:49.085402 15796 consensus_queue.cc:237] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260926 20:58:49.085821 15796 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a41d46a69bda4043a0b83db9c5a2fda5. Latest consensus state: current_term: 1 leader_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } } }
I20260926 20:58:49.086079 15796 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:58:49.086001 15797 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } } }
I20260926 20:58:49.086483 15797 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [sys.catalog]: This master's current role is: LEADER
W20260926 20:58:49.087388 15812 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:58:49.087451 15812 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:58:49.087510 15805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 20:58:49 dist-test-slave-fl7g krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:49.087733 15805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:58:49.089583 15805 catalog_manager.cc:1347] Generated new cluster ID: 323911ee9c1549a4ad23a6f298e828f9
I20260926 20:58:49.089649 15805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:58:49.091823 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.080617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.0.98.62:35223)
0926 20:58:49.080806 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:49.080817 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:49.080853 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:58:49.081300 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:49.081307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:49.081357 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:49.081472 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:49.082302 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.083212 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.084319 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.084927 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.085058 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.088223 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:49.088228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:49.088228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:49.088243 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:49.090219 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:49.090625 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:49.090628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:49.090629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:49.090730 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:49.090945 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:49.090946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:49.090946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:49.091203 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:49.091363 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:49.091475 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 20:58:49.091624 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":110,"threads_started":1}
I20260926 20:58:49.111747 15805 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:58:49.112353 15805 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:58:49.130434 15805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: Generated new TSK 0
I20260926 20:58:49.130698 15805 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:58:49.164337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35223
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:58:49.248123 15817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:49.248397 15817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:49.248466 15817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:49.249799 15817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:49.249866 15817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:49.249919 15817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:58:49.251596 15817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:49.251884 15817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:49.252133 15817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:49.257896 15825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:49.257862 15823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:49.257951 15817 server_base.cc:1034] running on GCE node
W20260926 20:58:49.257864 15822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:49.258251 15817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:49.258522 15817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:49.259760 15817 hybrid_clock.cc:648] HybridClock initialized: now 1790456329259745 us; error 47 us; skew 500 ppm
Sep 26 20:58:49 dist-test-slave-fl7g krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:49.262454 15817 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:58:49.262975 15817 webserver.cc:460] Webserver started at http://127.0.98.1:44195/ using document root <none> and password file <none>
I20260926 20:58:49.263171 15817 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:49.263214 15817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:49.263319 15817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:58:49.264338 15817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "19ed2f4f490244bf988a8f2f41dcb8f6"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "9f5c8299948bad12ea10cc2044e870f4"
server_key_iv: "d14a3101e2e9e4cfe69e0db859c1a2ca"
server_key_version: "encryptionkey@0"
I20260926 20:58:49.264734 15817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "19ed2f4f490244bf988a8f2f41dcb8f6"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "9f5c8299948bad12ea10cc2044e870f4"
server_key_iv: "d14a3101e2e9e4cfe69e0db859c1a2ca"
server_key_version: "encryptionkey@0"
I20260926 20:58:49.266041 15817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:49.266827 15832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:49.267004 15817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:49.267093 15817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "19ed2f4f490244bf988a8f2f41dcb8f6"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "9f5c8299948bad12ea10cc2044e870f4"
server_key_iv: "d14a3101e2e9e4cfe69e0db859c1a2ca"
server_key_version: "encryptionkey@0"
I20260926 20:58:49.267154 15817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:49.279912 15817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:49.282181 15817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:49.282335 15817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:49.282546 15817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:58:49.282889 15817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:58:49.282927 15817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:49.282950 15817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:58:49.282963 15817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:49.289453 15817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39583
I20260926 20:58:49.289531 15945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39583 every 8 connection(s)
I20260926 20:58:49.289882 15817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 20:58:49 dist-test-slave-fl7g krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:49.299551 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.290697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45039 (local address 127.0.98.62:35223)
0926 20:58:49.290791 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:49.290794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:49.291012 (+ 218us) server_negotiation.cc:407] Connection header received
0926 20:58:49.291476 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:49.291477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:49.291487 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:49.291533 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:49.292571 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.293084 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.294099 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.294309 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.294379 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.296659 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:49.296661 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:49.296661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:49.296670 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:49.298048 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:49.298521 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:49.298523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:49.298523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:49.298593 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:49.298862 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:49.298863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:49.298864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:49.298994 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:49.299074 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:49.299360 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 20:58:49.299455 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:58:49.299793 15948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.290670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:45039)
0926 20:58:49.290881 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:49.290919 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:49.291391 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:49.291618 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:49.291621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:49.292214 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:49.292476 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:49.292483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.293165 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.293166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:49.293988 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:49.293991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.294394 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.294395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:49.294499 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.295021 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:49.295029 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:49.296539 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:49.298176 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:49.298186 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:49.298189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:49.298437 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:49.298666 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:49.298667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:49.298668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:49.298791 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:49.299065 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:49.299072 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:49.299220 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 20:58:49.299340 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 20:58:49.299515 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":112,"threads_started":1}
I20260926 20:58:49.299979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 15817
I20260926 20:58:49.300097 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:58:49.300318 15946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35223
I20260926 20:58:49.300362 392 external_mini_cluster.cc:1428] Setting key b576a8b3bea18738c03ae60a6ec25ade
I20260926 20:58:49.300452 15946 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:49.300652 15946 heartbeater.cc:511] Master 127.0.98.62:35223 requested a full tablet report, sending...
I20260926 20:58:49.301224 15755 ts_manager.cc:194] Registered new tserver with Master: 19ed2f4f490244bf988a8f2f41dcb8f6 (127.0.98.1:39583)
I20260926 20:58:49.301853 15755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45039
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:58:49.328109 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35223
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:58:49.412344 15951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:49.412619 15951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:49.412647 15951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:49.413909 15951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:49.413967 15951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:49.413996 15951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:58:49.415529 15951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:49.415827 15951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:49.416119 15951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:49.421718 15959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:49.421718 15956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:49.421756 15957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:49.421994 15951 server_base.cc:1034] running on GCE node
I20260926 20:58:49.422216 15951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:49.422459 15951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:49.423796 15951 hybrid_clock.cc:648] HybridClock initialized: now 1790456329423770 us; error 33 us; skew 500 ppm
Sep 26 20:58:49 dist-test-slave-fl7g krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:49.426568 15951 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:58:49.427111 15951 webserver.cc:460] Webserver started at http://127.0.98.2:46443/ using document root <none> and password file <none>
I20260926 20:58:49.427372 15951 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:49.427443 15951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:49.427595 15951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:58:49.428812 15951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "2a69e1ea6db94c69bc99a1e2721a3669"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "fa55bd102089ba53e48c9fc27dfb2037"
server_key_iv: "cfd08c987b6a90e5aa90057405c0a14d"
server_key_version: "encryptionkey@0"
I20260926 20:58:49.429186 15951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2a69e1ea6db94c69bc99a1e2721a3669"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "fa55bd102089ba53e48c9fc27dfb2037"
server_key_iv: "cfd08c987b6a90e5aa90057405c0a14d"
server_key_version: "encryptionkey@0"
I20260926 20:58:49.430624 15951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:58:49.431490 15966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:49.431741 15951 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:49.431867 15951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "2a69e1ea6db94c69bc99a1e2721a3669"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "fa55bd102089ba53e48c9fc27dfb2037"
server_key_iv: "cfd08c987b6a90e5aa90057405c0a14d"
server_key_version: "encryptionkey@0"
I20260926 20:58:49.431972 15951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:49.452751 15951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:49.455688 15951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:49.455883 15951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:49.456172 15951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:58:49.456601 15951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:58:49.456662 15951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:49.456712 15951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:58:49.456748 15951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:49.463140 15951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34835
I20260926 20:58:49.463207 16079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34835 every 8 connection(s)
I20260926 20:58:49.463598 15951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:58:49.463907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 15951
I20260926 20:58:49.464027 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:58:49.464268 392 external_mini_cluster.cc:1428] Setting key d07f973a0aa39079cea6b5e857d10a1d
Sep 26 20:58:49 dist-test-slave-fl7g krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 20:58:49.475236 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.464467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36237 (local address 127.0.98.62:35223)
0926 20:58:49.464682 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:49.464684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:49.464812 (+ 128us) server_negotiation.cc:407] Connection header received
0926 20:58:49.465545 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:49.465546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:49.465556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:49.465608 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:49.467078 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.467878 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.469046 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.469404 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.469486 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.471832 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:49.471834 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:49.471834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:49.471843 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:49.473532 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:49.474020 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:49.474021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:49.474021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:49.474077 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:49.474346 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:49.474347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:49.474348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:49.474504 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:49.474630 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:49.475079 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 20:58:49.475164 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260926 20:58:49.475406 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.464410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:36237)
0926 20:58:49.464620 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:49.464675 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:49.465419 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:49.465694 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:49.465699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:49.466625 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:49.466962 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:49.466968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.467982 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.467983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:49.468949 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:49.468952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.469496 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.469497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:49.469628 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.470266 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:49.470274 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:49.471734 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:49.473626 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:49.473641 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:49.473643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:49.473934 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:49.474144 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:49.474145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:49.474145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:49.474251 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:49.474684 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:49.474690 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:49.474889 (+ 199us) client_negotiation.cc:769] Sending connection context
0926 20:58:49.475012 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 20:58:49.475161 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260926 20:58:49.476118 16080 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35223
I20260926 20:58:49.476257 16080 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 20:58:49.476514 16080 heartbeater.cc:511] Master 127.0.98.62:35223 requested a full tablet report, sending...
I20260926 20:58:49.477111 15755 ts_manager.cc:194] Registered new tserver with Master: 2a69e1ea6db94c69bc99a1e2721a3669 (127.0.98.2:34835)
I20260926 20:58:49.477648 15755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:58:49.491950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35223
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:58:49.577641 16085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:58:49.577965 16085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:58:49.578011 16085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:58:49.579284 16085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:58:49.579370 16085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:58:49.579419 16085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:58:49.581012 16085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:58:49.581310 16085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:58:49.581566 16085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:58:49.587007 16091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:49.587021 16090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:58:49.587044 16093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:58:49.587307 16085 server_base.cc:1034] running on GCE node
I20260926 20:58:49.587531 16085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:58:49.587795 16085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:58:49.588987 16085 hybrid_clock.cc:648] HybridClock initialized: now 1790456329588974 us; error 30 us; skew 500 ppm
Sep 26 20:58:49 dist-test-slave-fl7g krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:58:49.591967 16085 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:58:49.592624 16085 webserver.cc:460] Webserver started at http://127.0.98.3:42219/ using document root <none> and password file <none>
I20260926 20:58:49.592878 16085 fs_manager.cc:359] Metadata directory not provided
I20260926 20:58:49.592952 16085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:58:49.593129 16085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:58:49.594251 16085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "8f7e87e3a93a42589ca375b55acc1c51"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "dd257e2166d2996642375f4366bf90b5"
server_key_iv: "2fafac7838d0c695abcc23da0bc5faed"
server_key_version: "encryptionkey@0"
I20260926 20:58:49.594657 16085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8f7e87e3a93a42589ca375b55acc1c51"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "dd257e2166d2996642375f4366bf90b5"
server_key_iv: "2fafac7838d0c695abcc23da0bc5faed"
server_key_version: "encryptionkey@0"
I20260926 20:58:49.596105 16085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:58:49.597090 16100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:49.597348 16085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:58:49.597455 16085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "8f7e87e3a93a42589ca375b55acc1c51"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "dd257e2166d2996642375f4366bf90b5"
server_key_iv: "2fafac7838d0c695abcc23da0bc5faed"
server_key_version: "encryptionkey@0"
I20260926 20:58:49.597537 16085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:58:49.619138 16085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:58:49.621892 16085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:58:49.622076 16085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:58:49.622385 16085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:58:49.622757 16085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:58:49.622819 16085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:49.622867 16085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:58:49.622898 16085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:58:49.629649 16085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41145
I20260926 20:58:49.629694 16213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41145 every 8 connection(s)
I20260926 20:58:49.630057 16085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:58:49 dist-test-slave-fl7g krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:58:49.637650 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 16085
I20260926 20:58:49.637737 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:58:49.637964 392 external_mini_cluster.cc:1428] Setting key f70f540b4cf8b34c681d75694c95ba9f
I20260926 20:58:49.640923 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.630918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47077 (local address 127.0.98.62:35223)
0926 20:58:49.630996 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:49.630998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:49.631207 (+ 209us) server_negotiation.cc:407] Connection header received
0926 20:58:49.631761 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:49.631762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:49.631776 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:49.631827 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:49.632900 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.633416 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.634454 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.634662 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.634751 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.637358 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:49.637360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:49.637360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:49.637367 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:49.639289 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:49.639719 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:49.639721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:49.639721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:49.639783 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:49.640058 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:49.640059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:49.640059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:49.640184 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:49.640257 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:49.640706 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 20:58:49.640830 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:58:49.641117 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.630878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:47077)
0926 20:58:49.631070 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:49.631107 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:49.631638 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:49.631904 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:49.631908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:49.632582 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:58:49.632795 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:49.632800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.633518 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.633519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:49.634343 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:49.634345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.634764 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.634765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:49.634896 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.635546 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:58:49.635555 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:58:49.637249 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:58:49.639373 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:49.639382 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:49.639384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:49.639640 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:49.639861 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:58:49.639862 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:58:49.639862 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:58:49.639981 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:58:49.640261 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:58:49.640268 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:58:49.640457 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 20:58:49.640600 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 20:58:49.640864 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":102,"threads_started":1}
I20260926 20:58:49.641589 16214 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35223
I20260926 20:58:49.641676 16214 heartbeater.cc:464] Registering TS with master...
I20260926 20:58:49.641871 16214 heartbeater.cc:511] Master 127.0.98.62:35223 requested a full tablet report, sending...
I20260926 20:58:49.642262 15755 ts_manager.cc:194] Registered new tserver with Master: 8f7e87e3a93a42589ca375b55acc1c51 (127.0.98.3:41145)
I20260926 20:58:49.642745 15755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47077
I20260926 20:58:49.650094 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:58:49.657877 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.651582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51358 (local address 127.0.98.62:35223)
0926 20:58:49.651647 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:49.651649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:49.651780 (+ 131us) server_negotiation.cc:407] Connection header received
0926 20:58:49.651824 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:49.651824 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:49.651833 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:49.651889 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:58:49.652524 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.653038 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.653812 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.654007 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.654070 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.655312 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:58:49.655313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:58:49.655313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:58:49.655320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:58:49.656874 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:49.657191 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:49.657191 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:49.657192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:49.657244 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:58:49.657412 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:58:49.657412 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:58:49.657413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:58:49.657560 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:58:49.657636 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:49.657750 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 20:58:49.657810 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:58:49.659817 15755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:58:49.661621 15755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:58:49.668646 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.664220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39583 (local address 127.0.0.1:35190)
0926 20:58:49.664526 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:49.664538 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:49.664591 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:49.665203 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:49.665204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:49.665207 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:49.665248 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:49.665249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.666444 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.666445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:49.667406 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:49.667408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.668276 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.668277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:49.668365 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.668380 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 20:58:49.668470 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:58:49.668526 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":89,"threads_started":1}
I20260926 20:58:49.668746 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.664690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42146 (local address 127.0.98.3:41145)
0926 20:58:49.664857 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:49.664862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:49.665081 (+ 219us) server_negotiation.cc:407] Connection header received
0926 20:58:49.665125 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:49.665128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:49.665145 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:49.665292 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:49.665416 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.666100 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.667710 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.668391 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.668484 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.668541 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:49.668612 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 20:58:49.668665 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20260926 20:58:49.668814 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.664345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.0.98.2:34835)
0926 20:58:49.664667 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:49.664673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:49.664683 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:58:49.664730 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:49.664732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:49.664745 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:49.664839 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:49.665022 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.666168 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.667729 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.668207 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.668299 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.668360 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:49.668678 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 20:58:49.668725 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":90,"threads_started":1}
I20260926 20:58:49.668718 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.664193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34835 (local address 127.0.0.1:40580)
0926 20:58:49.664437 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:49.664486 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:49.664559 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:49.664845 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:49.664848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:49.664851 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:49.664933 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:49.664935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.666260 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.666261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:49.667620 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:49.667623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.668369 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.668371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:49.668480 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.668485 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:58:49.668534 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 20:58:49.668589 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":94,"thread_start_us":82,"threads_started":1}
I20260926 20:58:49.669274 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.664414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41145 (local address 127.0.0.1:42146)
0926 20:58:49.664962 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:49.664972 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:49.665021 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:49.665289 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:49.665289 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:49.665291 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:49.665349 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:49.665351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.666206 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.666207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:49.667621 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:49.667623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.668483 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.668484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:49.668536 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.668539 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:58:49.668598 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 20:58:49.668639 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":47,"thread_start_us":103,"threads_started":1}
I20260926 20:58:49.669519 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.664442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35190 (local address 127.0.98.1:39583)
0926 20:58:49.664977 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:49.664981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:49.664991 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:58:49.665029 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:49.665031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:49.665045 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:49.665117 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:49.665322 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.666347 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.667490 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.668178 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.668256 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.668315 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:49.669382 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0926 20:58:49.669435 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":98,"threads_started":1}
I20260926 20:58:49.669857 16014 tablet_service.cc:1467] Processing CreateTablet for tablet 1c87d341881b47abb8fe17a0c5515bd4 (DEFAULT_TABLE table=test-table [id=6e9ba68b2a0940c3a3dc5377e33248fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:58:49.669965 16148 tablet_service.cc:1467] Processing CreateTablet for tablet 1c87d341881b47abb8fe17a0c5515bd4 (DEFAULT_TABLE table=test-table [id=6e9ba68b2a0940c3a3dc5377e33248fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:58:49.670228 16014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c87d341881b47abb8fe17a0c5515bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:49.670250 16148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c87d341881b47abb8fe17a0c5515bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:49.670107 15880 tablet_service.cc:1467] Processing CreateTablet for tablet 1c87d341881b47abb8fe17a0c5515bd4 (DEFAULT_TABLE table=test-table [id=6e9ba68b2a0940c3a3dc5377e33248fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:58:49.670436 15880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c87d341881b47abb8fe17a0c5515bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:58:49.672955 16231 tablet_bootstrap.cc:492] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51: Bootstrap starting.
I20260926 20:58:49.673676 16231 tablet_bootstrap.cc:654] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51: Neither blocks nor log segments found. Creating new log.
I20260926 20:58:49.673691 16232 tablet_bootstrap.cc:492] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6: Bootstrap starting.
I20260926 20:58:49.673964 16233 tablet_bootstrap.cc:492] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669: Bootstrap starting.
I20260926 20:58:49.674353 16231 log.cc:824] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:49.674383 16232 tablet_bootstrap.cc:654] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6: Neither blocks nor log segments found. Creating new log.
I20260926 20:58:49.674611 16233 tablet_bootstrap.cc:654] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669: Neither blocks nor log segments found. Creating new log.
I20260926 20:58:49.674993 16232 log.cc:824] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:49.675259 16231 tablet_bootstrap.cc:492] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51: No bootstrap required, opened a new log
I20260926 20:58:49.675246 16233 log.cc:824] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669: Log is configured to *not* fsync() on all Append() calls
I20260926 20:58:49.675359 16231 ts_tablet_manager.cc:1397] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:58:49.675722 16232 tablet_bootstrap.cc:492] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6: No bootstrap required, opened a new log
I20260926 20:58:49.675796 16232 ts_tablet_manager.cc:1397] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:58:49.676019 16233 tablet_bootstrap.cc:492] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669: No bootstrap required, opened a new log
I20260926 20:58:49.676085 16233 ts_tablet_manager.cc:1397] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:58:49.677059 16231 raft_consensus.cc:348] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } }
I20260926 20:58:49.677232 16231 raft_consensus.cc:374] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:49.677277 16231 raft_consensus.cc:729] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f7e87e3a93a42589ca375b55acc1c51, State: Initialized, Role: FOLLOWER
I20260926 20:58:49.677408 16231 consensus_queue.cc:260] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } }
I20260926 20:58:49.677701 16231 ts_tablet_manager.cc:1428] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:58:49.677714 16214 heartbeater.cc:503] Master 127.0.98.62:35223 was elected leader, sending a full tablet report...
I20260926 20:58:49.677870 16232 raft_consensus.cc:348] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } }
I20260926 20:58:49.678051 16232 raft_consensus.cc:374] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:49.678088 16232 raft_consensus.cc:729] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ed2f4f490244bf988a8f2f41dcb8f6, State: Initialized, Role: FOLLOWER
I20260926 20:58:49.678205 16232 consensus_queue.cc:260] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } }
I20260926 20:58:49.678471 16232 ts_tablet_manager.cc:1428] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:58:49.678483 15946 heartbeater.cc:503] Master 127.0.98.62:35223 was elected leader, sending a full tablet report...
I20260926 20:58:49.678450 16233 raft_consensus.cc:348] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } }
I20260926 20:58:49.678613 16233 raft_consensus.cc:374] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:58:49.678659 16233 raft_consensus.cc:729] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a69e1ea6db94c69bc99a1e2721a3669, State: Initialized, Role: FOLLOWER
I20260926 20:58:49.678795 16233 consensus_queue.cc:260] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } }
I20260926 20:58:49.679107 16080 heartbeater.cc:503] Master 127.0.98.62:35223 was elected leader, sending a full tablet report...
I20260926 20:58:49.679090 16233 ts_tablet_manager.cc:1428] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:58:49.684603 16238 raft_consensus.cc:482] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:58:49.684713 16238 raft_consensus.cc:504] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } }
I20260926 20:58:49.685145 16238 leader_election.cc:288] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a69e1ea6db94c69bc99a1e2721a3669 (127.0.98.2:34835), 8f7e87e3a93a42589ca375b55acc1c51 (127.0.98.3:41145)
I20260926 20:58:49.688342 15948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.685375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41145 (local address 127.0.98.1:43859)
0926 20:58:49.685431 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:49.685447 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:49.685508 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:49.685872 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:49.685873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:49.685874 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:49.685966 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:49.685968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.686661 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.686663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:49.687465 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:49.687467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.688112 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.688113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:49.688192 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.688197 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:58:49.688251 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:58:49.688288 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:58:49.688450 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.685451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43859 (local address 127.0.98.3:41145)
0926 20:58:49.685499 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:49.685502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:49.685514 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:58:49.685656 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:49.685657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:49.685670 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:49.685742 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:49.686066 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.686579 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.687559 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.688022 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.688088 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.688097 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:49.688286 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 20:58:49.688337 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:58:49.688544 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.685375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34835 (local address 127.0.98.1:57409)
0926 20:58:49.685597 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:49.685607 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:58:49.685681 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:58:49.685835 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:58:49.685836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:58:49.685839 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:58:49.685923 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:49.685925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.686720 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.686720 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:49.687618 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:58:49.687621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.688220 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.688221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:58:49.688325 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.688329 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:58:49.688397 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:58:49.688457 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":59,"thread_start_us":101,"threads_started":1}
I20260926 20:58:49.688558 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.685572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57409 (local address 127.0.98.2:34835)
0926 20:58:49.685618 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:49.685619 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:49.685654 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:58:49.685752 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:49.685753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:49.685765 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:49.685820 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:58:49.686042 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.686608 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.687713 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.688107 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.688180 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.688190 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:49.688435 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 20:58:49.688483 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 20:58:49.688862 16034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c87d341881b47abb8fe17a0c5515bd4" candidate_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" is_pre_election: true
I20260926 20:58:49.689010 16034 raft_consensus.cc:2454] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19ed2f4f490244bf988a8f2f41dcb8f6 in term 0.
I20260926 20:58:49.689145 16168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c87d341881b47abb8fe17a0c5515bd4" candidate_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f7e87e3a93a42589ca375b55acc1c51" is_pre_election: true
I20260926 20:58:49.689253 16168 raft_consensus.cc:2454] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19ed2f4f490244bf988a8f2f41dcb8f6 in term 0.
I20260926 20:58:49.689235 15834 leader_election.cc:302] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19ed2f4f490244bf988a8f2f41dcb8f6, 2a69e1ea6db94c69bc99a1e2721a3669; no voters:
I20260926 20:58:49.689437 16238 raft_consensus.cc:2781] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:58:49.689517 16238 raft_consensus.cc:482] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:58:49.689553 16238 raft_consensus.cc:3037] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:49.690267 16238 raft_consensus.cc:504] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } }
I20260926 20:58:49.690421 16238 leader_election.cc:288] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 1 election: Requested vote from peers 2a69e1ea6db94c69bc99a1e2721a3669 (127.0.98.2:34835), 8f7e87e3a93a42589ca375b55acc1c51 (127.0.98.3:41145)
I20260926 20:58:49.690577 16168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c87d341881b47abb8fe17a0c5515bd4" candidate_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f7e87e3a93a42589ca375b55acc1c51"
I20260926 20:58:49.690627 16034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c87d341881b47abb8fe17a0c5515bd4" candidate_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a69e1ea6db94c69bc99a1e2721a3669"
I20260926 20:58:49.690680 16168 raft_consensus.cc:3037] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:49.690717 16034 raft_consensus.cc:3037] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:58:49.691447 16034 raft_consensus.cc:2454] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19ed2f4f490244bf988a8f2f41dcb8f6 in term 1.
I20260926 20:58:49.691582 16168 raft_consensus.cc:2454] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19ed2f4f490244bf988a8f2f41dcb8f6 in term 1.
I20260926 20:58:49.691636 15834 leader_election.cc:302] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19ed2f4f490244bf988a8f2f41dcb8f6, 2a69e1ea6db94c69bc99a1e2721a3669; no voters:
I20260926 20:58:49.691773 16238 raft_consensus.cc:2781] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:58:49.691954 16238 raft_consensus.cc:686] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 LEADER]: Becoming Leader. State: Replica: 19ed2f4f490244bf988a8f2f41dcb8f6, State: Running, Role: LEADER
I20260926 20:58:49.692057 16238 consensus_queue.cc:237] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } }
I20260926 20:58:49.692883 15755 catalog_manager.cc:5495] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19ed2f4f490244bf988a8f2f41dcb8f6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260926 20:58:49.705111 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.702481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35206 (local address 127.0.98.1:39583)
0926 20:58:49.702551 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 20:58:49.702553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:58:49.702564 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:58:49.702616 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:58:49.702616 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:58:49.702638 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:58:49.702705 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:58:49.702829 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.703335 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.704101 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:58:49.704301 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:58:49.704369 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:58:49.704685 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:58:49.704792 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:58:49.704872 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:58:49.704971 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 20:58:49.704992 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:58:49.706992 16034 raft_consensus.cc:1261] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 1 FOLLOWER]: Refusing update from remote peer 19ed2f4f490244bf988a8f2f41dcb8f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:58:49.707003 16168 raft_consensus.cc:1261] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51 [term 1 FOLLOWER]: Refusing update from remote peer 19ed2f4f490244bf988a8f2f41dcb8f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:58:49.707409 16241 consensus_queue.cc:1035] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:58:49.707548 16238 consensus_queue.cc:1035] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:58:49.711272 16244 mvcc.cc:205] Tried to move back new op lower bound from 7333709126477045760 to 7333709126419365888. Current Snapshot: MvccSnapshot[applied={T|T < 7333709126477045760 or (T in {7333709126477045760})}]
I20260926 20:58:49.713439 15755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:58:49.713567 15755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:58:49.714429 15755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: Sending DeleteTablet for 3 replicas of tablet 1c87d341881b47abb8fe17a0c5515bd4
I20260926 20:58:49.714798 16014 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c87d341881b47abb8fe17a0c5515bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:58:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40580
I20260926 20:58:49.714884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 15817
I20260926 20:58:49.714865 15880 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c87d341881b47abb8fe17a0c5515bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:58:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35190
I20260926 20:58:49.715060 16148 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c87d341881b47abb8fe17a0c5515bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:58:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42146
I20260926 20:58:49.715715 16255 tablet_replica.cc:330] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51: stopping tablet replica
I20260926 20:58:49.715832 16255 raft_consensus.cc:2229] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:58:49.715986 16255 raft_consensus.cc:2258] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:58:49.716177 16254 tablet_replica.cc:330] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669: stopping tablet replica
I20260926 20:58:49.716310 16254 raft_consensus.cc:2229] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:58:49.716441 16254 raft_consensus.cc:2258] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:58:49.716686 16255 ts_tablet_manager.cc:1905] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:58:49.716928 16254 ts_tablet_manager.cc:1905] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:58:49.718700 16255 ts_tablet_manager.cc:1918] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:58:49.718791 16255 log.cc:1196] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/1c87d341881b47abb8fe17a0c5515bd4
I20260926 20:58:49.718883 16254 ts_tablet_manager.cc:1918] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:58:49.718954 16254 log.cc:1196] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/1c87d341881b47abb8fe17a0c5515bd4
I20260926 20:58:49.719118 16255 ts_tablet_manager.cc:1939] T 1c87d341881b47abb8fe17a0c5515bd4 P 8f7e87e3a93a42589ca375b55acc1c51: Deleting consensus metadata
I20260926 20:58:49.719216 16254 ts_tablet_manager.cc:1939] T 1c87d341881b47abb8fe17a0c5515bd4 P 2a69e1ea6db94c69bc99a1e2721a3669: Deleting consensus metadata
I20260926 20:58:49.719564 15739 catalog_manager.cc:4841] TS 8f7e87e3a93a42589ca375b55acc1c51 (127.0.98.3:41145): tablet 1c87d341881b47abb8fe17a0c5515bd4 (table test-table [id=6e9ba68b2a0940c3a3dc5377e33248fa]) successfully deleted
I20260926 20:58:49.719568 15740 catalog_manager.cc:4841] TS 2a69e1ea6db94c69bc99a1e2721a3669 (127.0.98.2:34835): tablet 1c87d341881b47abb8fe17a0c5515bd4 (table test-table [id=6e9ba68b2a0940c3a3dc5377e33248fa]) successfully deleted
W20260926 20:58:49.720831 15739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39583 (error 108)
I20260926 20:58:49.721419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 15951
I20260926 20:58:49.721488 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:58:49.721344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39583 (local address 127.0.0.1:35220)
0926 20:58:49.721408 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 20:58:49.721448 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 20:58:49.722160 15739 catalog_manager.cc:4584] TS 19ed2f4f490244bf988a8f2f41dcb8f6 (127.0.98.1:39583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c87d341881b47abb8fe17a0c5515bd4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39583: connect: Connection refused (error 111)
I20260926 20:58:49.727250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 16085
I20260926 20:58:49.732261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 15723
I20260926 20:59:09.737923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35223
--webserver_interface=127.0.98.62
--webserver_port=41187
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:09.821839 16257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:09.822078 16257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:09.822129 16257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:09.823524 16257 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:59:09.823583 16257 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:59:09.823598 16257 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:59:09.823611 16257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:59:09.823624 16257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:09.823684 16257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:09.823702 16257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:59:09.823719 16257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:59:09.825343 16257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:09.825629 16257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:09.825906 16257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:59:09.836010 16257 server_base.cc:1034] running on GCE node
W20260926 20:59:09.836014 16264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:09.836267 16266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:09.836014 16263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:09.836696 16257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:09.837064 16257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:09.838249 16257 hybrid_clock.cc:648] HybridClock initialized: now 1790456349838227 us; error 41 us; skew 500 ppm
Sep 26 20:59:09 dist-test-slave-fl7g krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:09.847519 16257 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:59:09.848109 16257 webserver.cc:460] Webserver started at http://127.0.98.62:41187/ using document root <none> and password file <none>
I20260926 20:59:09.848400 16257 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:09.848542 16257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:09.850209 16257 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:59:09.851075 16273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:09.851336 16257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:59:09.851459 16257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "a41d46a69bda4043a0b83db9c5a2fda5"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "921261bbb787c3fa6c9527d7f1498290"
server_key_iv: "907bec97c4f824e8a6d6bfac27f166eb"
server_key_version: "encryptionkey@0"
I20260926 20:59:09.851943 16257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:09.872992 16257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:09.875217 16257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:09.875403 16257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:09.879743 16257 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35223
I20260926 20:59:09.879776 16325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35223 every 8 connection(s)
I20260926 20:59:09.880208 16257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:59:09.881898 16326 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:59:09.882750 16326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: Bootstrap starting.
I20260926 20:59:09.883806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 16257
I20260926 20:59:09.884116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39583
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35223
--webserver_port=44195
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:59:09.891687 16326 log.cc:824] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:09.893870 16326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:59:09.894192 16326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: Bootstrap complete.
I20260926 20:59:09.896677 16326 raft_consensus.cc:348] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260926 20:59:09.896960 16326 raft_consensus.cc:729] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41d46a69bda4043a0b83db9c5a2fda5, State: Initialized, Role: FOLLOWER
I20260926 20:59:09.897150 16326 consensus_queue.cc:260] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260926 20:59:09.897233 16326 raft_consensus.cc:388] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:59:09.897315 16326 raft_consensus.cc:482] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:59:09.897368 16326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:59:09.898419 16326 raft_consensus.cc:504] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260926 20:59:09.898553 16326 leader_election.cc:302] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a41d46a69bda4043a0b83db9c5a2fda5; no voters:
I20260926 20:59:09.898778 16326 leader_election.cc:288] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:59:09.898818 16330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:59:09.898936 16330 raft_consensus.cc:686] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 2 LEADER]: Becoming Leader. State: Replica: a41d46a69bda4043a0b83db9c5a2fda5, State: Running, Role: LEADER
I20260926 20:59:09.899076 16326 sys_catalog.cc:564] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:59:09.899092 16330 consensus_queue.cc:237] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260926 20:59:09.899410 16330 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a41d46a69bda4043a0b83db9c5a2fda5. Latest consensus state: current_term: 2 leader_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } } }
I20260926 20:59:09.899482 16330 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:09.899598 16331 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } } }
I20260926 20:59:09.899704 16331 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:09.900327 16341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:59:09.901155 16341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:59:09.901263 16341 catalog_manager.cc:1259] Loaded cluster ID: 323911ee9c1549a4ad23a6f298e828f9
I20260926 20:59:09.901298 16341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:59:09.902573 16341 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:59:09.902807 16341 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: Loaded TSK: 0
I20260926 20:59:09.919667 16341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: Generated new TSK 1
I20260926 20:59:09.919811 16341 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:59:09.974834 16328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:09.975069 16328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:09.975096 16328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:09.976441 16328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:09.976537 16328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:09.976586 16328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:59:09.978181 16328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44195
--tserver_master_addrs=127.0.98.62:35223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:09.978494 16328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:09.978760 16328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:09.984344 16356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:09.984357 16353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:09.984611 16328 server_base.cc:1034] running on GCE node
W20260926 20:59:09.984344 16354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:09.984939 16328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:09.985237 16328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:09.986464 16328 hybrid_clock.cc:648] HybridClock initialized: now 1790456349986448 us; error 35 us; skew 500 ppm
Sep 26 20:59:09 dist-test-slave-fl7g krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:09.989521 16328 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:59:09.990150 16328 webserver.cc:460] Webserver started at http://127.0.98.1:44195/ using document root <none> and password file <none>
I20260926 20:59:09.990406 16328 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:09.990497 16328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:09.992024 16328 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:59:09.992893 16363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:09.993065 16328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:59:09.993170 16328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "19ed2f4f490244bf988a8f2f41dcb8f6"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "9f5c8299948bad12ea10cc2044e870f4"
server_key_iv: "d14a3101e2e9e4cfe69e0db859c1a2ca"
server_key_version: "encryptionkey@0"
I20260926 20:59:09.993521 16328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:10.016125 16328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:10.018393 16328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:10.018586 16328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:10.018882 16328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:10.019353 16370 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:59:10.021374 16328 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:59:10.021453 16328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 20:59:10.021500 16328 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:59:10.022207 16328 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:59:10.022270 16328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 20:59:10.022338 16370 tablet_bootstrap.cc:492] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6: Bootstrap starting.
I20260926 20:59:10.029917 16328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39583
I20260926 20:59:10.030076 16477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39583 every 8 connection(s)
I20260926 20:59:10.030390 16328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:59:10.033524 16370 log.cc:824] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:10.036092 16370 tablet_bootstrap.cc:492] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:59:10.036517 16370 tablet_bootstrap.cc:492] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6: Bootstrap complete.
I20260926 20:59:10.037772 16370 ts_tablet_manager.cc:1397] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.010s
I20260926 20:59:10.039301 16370 raft_consensus.cc:348] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } }
I20260926 20:59:10.039577 16370 raft_consensus.cc:729] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ed2f4f490244bf988a8f2f41dcb8f6, State: Initialized, Role: FOLLOWER
I20260926 20:59:10.039719 16370 consensus_queue.cc:260] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } }
Sep 26 20:59:10 dist-test-slave-fl7g krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:59:10.039881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 16328
I20260926 20:59:10.040040 16370 ts_tablet_manager.cc:1428] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 20:59:10.040169 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34835
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35223
--webserver_port=46443
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:59:10.044914 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.031505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38283 (local address 127.0.98.62:35223)
0926 20:59:10.032611 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:10.032620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:10.032675 (+ 55us) server_negotiation.cc:407] Connection header received
0926 20:59:10.032798 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:10.032803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:10.032849 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:10.033024 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:10.033919 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.035308 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.036944 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.037548 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.037677 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.040678 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:10.040683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:10.040684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:10.040697 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:10.042965 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:10.043403 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:10.043407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:10.043408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:10.043539 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:10.043788 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:10.043789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:10.043790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:10.044154 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:10.044252 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:10.044585 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 20:59:10.044713 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":87,"threads_started":1}
I20260926 20:59:10.044919 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.031190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:38283)
0926 20:59:10.031397 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:10.031444 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:10.032021 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:10.033067 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:10.033071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:10.033471 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:10.033764 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:10.033773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.035464 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.035466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:10.036845 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:10.036850 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.037678 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.037680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:10.037842 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.038487 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:59:10.038497 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:10.040543 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:10.043064 (+ 2521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:10.043071 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:10.043073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:10.043308 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:10.043612 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:10.043613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:10.043613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:10.043711 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:10.044234 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:10.044242 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:10.044364 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 20:59:10.044509 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 20:59:10.044702 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":116,"threads_started":1}
I20260926 20:59:10.045648 16478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35223
I20260926 20:59:10.045740 16478 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:10.045938 16478 heartbeater.cc:511] Master 127.0.98.62:35223 requested a full tablet report, sending...
I20260926 20:59:10.046639 16290 ts_manager.cc:194] Registered new tserver with Master: 19ed2f4f490244bf988a8f2f41dcb8f6 (127.0.98.1:39583)
I20260926 20:59:10.047663 16290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38283
I20260926 20:59:10.048645 16478 heartbeater.cc:503] Master 127.0.98.62:35223 was elected leader, sending a full tablet report...
I20260926 20:59:10.051855 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.047240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39583 (local address 127.0.0.1:60224)
0926 20:59:10.048530 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:10.048576 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:10.048650 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:10.048967 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:10.048971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:10.048973 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:10.049044 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:10.049046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.049915 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.049917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:10.050856 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:10.050859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.051501 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.051502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:10.051585 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.051598 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 20:59:10.051696 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 20:59:10.051777 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1254,"thread_start_us":98,"threads_started":1}
I20260926 20:59:10.051882 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.047311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.0.98.1:39583)
0926 20:59:10.048796 (+ 1485us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:10.048800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:10.048810 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:59:10.048848 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:10.048850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:10.048866 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:10.048951 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:10.049114 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.049825 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.050947 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.051408 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.051500 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.051560 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:10.051713 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 20:59:10.051816 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1448,"thread_start_us":89,"threads_started":1}
I20260926 20:59:10.052337 16412 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c87d341881b47abb8fe17a0c5515bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:58:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60224
I20260926 20:59:10.052942 16487 tablet_replica.cc:330] stopping tablet replica
I20260926 20:59:10.053061 16487 raft_consensus.cc:2229] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:59:10.053130 16487 raft_consensus.cc:2258] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:59:10.053455 16487 ts_tablet_manager.cc:1905] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:59:10.056233 16487 ts_tablet_manager.cc:1918] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:59:10.056329 16487 log.cc:1196] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/wal/wals/1c87d341881b47abb8fe17a0c5515bd4
I20260926 20:59:10.056607 16487 ts_tablet_manager.cc:1939] T 1c87d341881b47abb8fe17a0c5515bd4 P 19ed2f4f490244bf988a8f2f41dcb8f6: Deleting consensus metadata
I20260926 20:59:10.057173 16274 catalog_manager.cc:4841] TS 19ed2f4f490244bf988a8f2f41dcb8f6 (127.0.98.1:39583): tablet 1c87d341881b47abb8fe17a0c5515bd4 (table test-table [id=6e9ba68b2a0940c3a3dc5377e33248fa]) successfully deleted
W20260926 20:59:10.128595 16484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:10.128847 16484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:10.128875 16484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:10.130194 16484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:10.130249 16484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:10.130316 16484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:59:10.131916 16484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46443
--tserver_master_addrs=127.0.98.62:35223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:10.132203 16484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:10.132513 16484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:10.137921 16492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:10.137921 16495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:10.137917 16493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:10.138128 16484 server_base.cc:1034] running on GCE node
I20260926 20:59:10.138358 16484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:10.138654 16484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:10.139860 16484 hybrid_clock.cc:648] HybridClock initialized: now 1790456350139841 us; error 27 us; skew 500 ppm
Sep 26 20:59:10 dist-test-slave-fl7g krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:10.143124 16484 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:59:10.143667 16484 webserver.cc:460] Webserver started at http://127.0.98.2:46443/ using document root <none> and password file <none>
I20260926 20:59:10.143934 16484 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:10.144023 16484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:10.145569 16484 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:10.146247 16502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:10.146435 16484 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:10.146543 16484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "2a69e1ea6db94c69bc99a1e2721a3669"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "fa55bd102089ba53e48c9fc27dfb2037"
server_key_iv: "cfd08c987b6a90e5aa90057405c0a14d"
server_key_version: "encryptionkey@0"
I20260926 20:59:10.146947 16484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:10.183642 16484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:10.185864 16484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:10.186038 16484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:10.186329 16484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:10.186661 16484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:10.186724 16484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:10.186759 16484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:10.186787 16484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:10.193111 16484 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34835
I20260926 20:59:10.193154 16615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34835 every 8 connection(s)
I20260926 20:59:10.193509 16484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:59:10.196537 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 16484
I20260926 20:59:10.196887 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41145
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35223
--webserver_port=42219
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:59:10 dist-test-slave-fl7g krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:59:10.203944 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.194218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60967 (local address 127.0.98.62:35223)
0926 20:59:10.194302 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:10.194304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:10.194459 (+ 155us) server_negotiation.cc:407] Connection header received
0926 20:59:10.194982 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:10.194983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:10.194998 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:10.195052 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:10.195998 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.196735 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.197851 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.198172 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.198237 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.200958 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:10.200960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:10.200960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:10.200967 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:10.202452 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:10.202908 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:10.202910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:10.202910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:10.202973 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:10.203238 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:10.203239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:10.203240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:10.203361 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:10.203435 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:10.203744 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 20:59:10.203844 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:59:10.204155 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.194155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:60967)
0926 20:59:10.194307 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:10.194347 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:10.194889 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:10.195110 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:10.195114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:10.195711 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:10.195916 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:10.195920 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.196829 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.196830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:10.197755 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:10.197758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.198271 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.198272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:10.198469 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.199034 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:59:10.199046 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:10.200817 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:10.202560 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:10.202569 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:10.202571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:10.202813 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:10.203045 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:10.203046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:10.203047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:10.203168 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:10.203438 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:10.203445 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:10.203594 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 20:59:10.203722 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 20:59:10.203897 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":90,"threads_started":1}
I20260926 20:59:10.204666 16616 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35223
I20260926 20:59:10.204764 16616 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:10.204970 16616 heartbeater.cc:511] Master 127.0.98.62:35223 requested a full tablet report, sending...
I20260926 20:59:10.205397 16290 ts_manager.cc:194] Registered new tserver with Master: 2a69e1ea6db94c69bc99a1e2721a3669 (127.0.98.2:34835)
I20260926 20:59:10.205986 16290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60967
W20260926 20:59:10.281597 16619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:10.281860 16619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:10.281890 16619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:10.283234 16619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:10.283326 16619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:10.283399 16619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:59:10.285055 16619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42219
--tserver_master_addrs=127.0.98.62:35223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:10.285362 16619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:10.285625 16619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:10.291136 16625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:10.291213 16619 server_base.cc:1034] running on GCE node
W20260926 20:59:10.291147 16624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:10.291138 16627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:10.291582 16619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:10.291898 16619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:10.293093 16619 hybrid_clock.cc:648] HybridClock initialized: now 1790456350293078 us; error 32 us; skew 500 ppm
Sep 26 20:59:10 dist-test-slave-fl7g krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:10.295889 16619 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:59:10.296490 16619 webserver.cc:460] Webserver started at http://127.0.98.3:42219/ using document root <none> and password file <none>
I20260926 20:59:10.296801 16619 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:10.296902 16619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:10.298426 16619 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:59:10.299480 16634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:10.299685 16619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:10.299803 16619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "8f7e87e3a93a42589ca375b55acc1c51"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "dd257e2166d2996642375f4366bf90b5"
server_key_iv: "2fafac7838d0c695abcc23da0bc5faed"
server_key_version: "encryptionkey@0"
I20260926 20:59:10.300258 16619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:10.320638 16619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:10.324265 16619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:10.324491 16619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:10.324784 16619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:10.325228 16619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:10.325282 16619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:10.325320 16619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:10.325346 16619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:10.333717 16619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41145
I20260926 20:59:10.333977 16747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41145 every 8 connection(s)
I20260926 20:59:10.334280 16619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:59:10 dist-test-slave-fl7g krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:59:10.342912 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 16619
I20260926 20:59:10.347940 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.335242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:49983)
0926 20:59:10.335411 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:10.335456 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:10.336493 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:10.336956 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:10.336961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:10.337379 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:10.337710 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:10.337717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.338872 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.338873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:10.340071 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:10.340074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.340600 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.340601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:10.340776 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.341425 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:59:10.341438 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:10.343826 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:10.345872 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:10.345884 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:10.345887 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:10.346259 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:10.346570 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:10.346571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:10.346571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:10.346746 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:10.347119 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:10.347127 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:10.347333 (+ 206us) client_negotiation.cc:769] Sending connection context
0926 20:59:10.347463 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 20:59:10.347669 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260926 20:59:10.348065 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.335448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49983 (local address 127.0.98.62:35223)
0926 20:59:10.335516 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:10.335518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:10.335531 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:59:10.336751 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:10.336753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:10.336768 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:10.336840 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:10.337841 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.338755 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.340194 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.340484 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.340561 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.343983 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:10.343985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:10.343985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:10.343993 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:10.345780 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:10.346380 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:10.346382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:10.346382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:10.346476 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:10.346862 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:10.346863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:10.346864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:10.347030 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:10.347113 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:10.347829 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 20:59:10.347952 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:59:10.348598 16748 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35223
I20260926 20:59:10.348733 16748 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:10.348975 16748 heartbeater.cc:511] Master 127.0.98.62:35223 requested a full tablet report, sending...
I20260926 20:59:10.349428 16290 ts_manager.cc:194] Registered new tserver with Master: 8f7e87e3a93a42589ca375b55acc1c51 (127.0.98.3:41145)
I20260926 20:59:10.349970 16290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49983
I20260926 20:59:10.351856 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.343508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.0.98.62:35223)
0926 20:59:10.343626 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:10.343627 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:10.343637 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:59:10.343746 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:10.343747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:10.343761 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:10.343818 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:10.344619 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.345717 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.346818 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.347056 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.347136 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.348777 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:10.348779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:10.348779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:10.348786 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:10.350508 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:10.350890 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:10.350891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:10.350892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:10.350956 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:10.351167 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:10.351168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:10.351169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:10.351285 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:10.351575 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:10.351619 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 20:59:10.351712 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":49,"threads_started":1}
I20260926 20:59:10.353080 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:59:10.356979 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.353450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35822 (local address 127.0.98.62:35223)
0926 20:59:10.353492 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:10.353494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:10.353636 (+ 142us) server_negotiation.cc:407] Connection header received
0926 20:59:10.353714 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:10.353714 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:10.353745 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:10.353836 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:59:10.353989 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.354817 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.356043 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.356340 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.356469 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.356659 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:59:10.356719 (+ 60us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 20:59:10.356914 (+ 195us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35822: token expired
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 20:59:10.357045 16752 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35822: token expired
W20260926 20:59:10.357069 16219 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:35223)
I20260926 20:59:10.365911 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.357398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35830 (local address 127.0.98.62:35223)
0926 20:59:10.357433 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:10.357436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:10.357445 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:59:10.357490 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:10.357491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:10.357500 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:10.357571 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:10.358277 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.359058 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.360220 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.360550 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.360614 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.362365 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:10.362367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:10.362367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:10.362377 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:10.364338 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:10.364795 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:10.364796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:10.364796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:10.364871 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:10.365207 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:10.365208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:10.365208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:10.365451 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:10.365578 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:10.365661 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 20:59:10.365778 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260926 20:59:10.367493 16290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:59:10.370519 16290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:59:10.373185 16412 tablet_service.cc:1467] Processing CreateTablet for tablet 9aeb999083764dfa9232eebf0e9c6ce8 (DEFAULT_TABLE table=test-table [id=e579fbba16a84bc7a436b86064e6d0fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:10.373426 16412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aeb999083764dfa9232eebf0e9c6ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:10.374855 16370 tablet_bootstrap.cc:492] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6: Bootstrap starting.
I20260926 20:59:10.375222 16370 tablet_bootstrap.cc:654] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:10.376176 16370 tablet_bootstrap.cc:492] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6: No bootstrap required, opened a new log
I20260926 20:59:10.376246 16370 ts_tablet_manager.cc:1397] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:59:10.376433 16370 raft_consensus.cc:348] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } } peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260926 20:59:10.376518 16370 raft_consensus.cc:374] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:10.376588 16370 raft_consensus.cc:729] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ed2f4f490244bf988a8f2f41dcb8f6, State: Initialized, Role: FOLLOWER
I20260926 20:59:10.376652 16370 consensus_queue.cc:260] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } } peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260926 20:59:10.376787 16370 ts_tablet_manager.cc:1428] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:59:10.377368 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.372741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41145 (local address 127.0.0.1:60774)
0926 20:59:10.372858 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:10.372871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:10.372908 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:10.373230 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:10.373231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:10.373233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:10.373288 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:10.373290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.374937 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.374955 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:10.376092 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:10.376095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.377146 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.377147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:10.377205 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.377210 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:59:10.377241 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 20:59:10.377293 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 20:59:10.377843 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.372843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60774 (local address 127.0.98.3:41145)
0926 20:59:10.373030 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:10.373035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:10.373043 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:59:10.373077 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:10.373079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:10.373092 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:10.373365 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:10.373455 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.374799 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.376202 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.377036 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.377586 (+ 550us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.377677 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:10.377708 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 20:59:10.377755 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":103,"threads_started":1}
I20260926 20:59:10.379024 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.372802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34835 (local address 127.0.0.1:48366)
0926 20:59:10.373180 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:10.373191 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:10.373257 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:10.373734 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:10.373735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:10.373737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:10.373781 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:10.373782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.375437 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.375438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:10.377053 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:10.377056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.378739 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.378740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:10.378813 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.378818 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:59:10.378884 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 20:59:10.378933 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":65,"threads_started":1}
I20260926 20:59:10.378989 16682 tablet_service.cc:1467] Processing CreateTablet for tablet 9aeb999083764dfa9232eebf0e9c6ce8 (DEFAULT_TABLE table=test-table [id=e579fbba16a84bc7a436b86064e6d0fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:10.379411 16682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aeb999083764dfa9232eebf0e9c6ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:10.379539 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.373026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.0.98.2:34835)
0926 20:59:10.373208 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:10.373213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:10.373243 (+ 30us) server_negotiation.cc:407] Connection header received
0926 20:59:10.373462 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:10.373528 (+ 66us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:10.373546 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:10.373628 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:10.373863 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.375339 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.377161 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.378286 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.378369 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.378683 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:10.379336 (+ 653us) server_negotiation.cc:299] Negotiation successful
0926 20:59:10.379392 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":75,"threads_started":1}
I20260926 20:59:10.380483 16550 tablet_service.cc:1467] Processing CreateTablet for tablet 9aeb999083764dfa9232eebf0e9c6ce8 (DEFAULT_TABLE table=test-table [id=e579fbba16a84bc7a436b86064e6d0fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:10.380847 16550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aeb999083764dfa9232eebf0e9c6ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:10.383419 16758 tablet_bootstrap.cc:492] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51: Bootstrap starting.
I20260926 20:59:10.384069 16758 tablet_bootstrap.cc:654] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:10.384436 16759 tablet_bootstrap.cc:492] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669: Bootstrap starting.
I20260926 20:59:10.385133 16758 log.cc:824] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:10.385182 16759 tablet_bootstrap.cc:654] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:10.385782 16759 log.cc:824] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:10.385985 16758 tablet_bootstrap.cc:492] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51: No bootstrap required, opened a new log
I20260926 20:59:10.386060 16758 ts_tablet_manager.cc:1397] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:59:10.386587 16759 tablet_bootstrap.cc:492] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669: No bootstrap required, opened a new log
I20260926 20:59:10.386659 16759 ts_tablet_manager.cc:1397] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:59:10.388161 16758 raft_consensus.cc:348] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } } peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260926 20:59:10.388374 16758 raft_consensus.cc:374] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:10.388448 16758 raft_consensus.cc:729] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f7e87e3a93a42589ca375b55acc1c51, State: Initialized, Role: FOLLOWER
I20260926 20:59:10.388592 16758 consensus_queue.cc:260] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } } peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260926 20:59:10.388684 16759 raft_consensus.cc:348] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } } peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260926 20:59:10.388862 16759 raft_consensus.cc:374] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:10.388911 16759 raft_consensus.cc:729] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a69e1ea6db94c69bc99a1e2721a3669, State: Initialized, Role: FOLLOWER
I20260926 20:59:10.388926 16758 ts_tablet_manager.cc:1428] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:59:10.389053 16759 consensus_queue.cc:260] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } } peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260926 20:59:10.389112 16748 heartbeater.cc:503] Master 127.0.98.62:35223 was elected leader, sending a full tablet report...
I20260926 20:59:10.389350 16759 ts_tablet_manager.cc:1428] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:59:10.389401 16616 heartbeater.cc:503] Master 127.0.98.62:35223 was elected leader, sending a full tablet report...
W20260926 20:59:10.444123 16617 tablet.cc:2369] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:59:10.528649 16763 raft_consensus.cc:482] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:59:10.528848 16763 raft_consensus.cc:504] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } } peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260926 20:59:10.529311 16763 leader_election.cc:288] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f7e87e3a93a42589ca375b55acc1c51 (127.0.98.3:41145), 19ed2f4f490244bf988a8f2f41dcb8f6 (127.0.98.1:39583)
I20260926 20:59:10.533463 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.529662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41661 (local address 127.0.98.3:41145)
0926 20:59:10.529714 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:10.529716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:10.529729 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:59:10.529846 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:10.529848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:10.529867 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:10.529920 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:10.530103 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.530868 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.532353 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.532994 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.533075 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.533088 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:10.533345 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 20:59:10.533389 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 20:59:10.533463 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.529756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36853 (local address 127.0.98.1:39583)
0926 20:59:10.529806 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:10.529808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:10.529996 (+ 188us) server_negotiation.cc:407] Connection header received
0926 20:59:10.530026 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:10.530027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:10.530034 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:10.530108 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:10.530241 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.530957 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.532354 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.532994 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.533075 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.533087 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:10.533345 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 20:59:10.533389 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:59:10.533475 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.529552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41145 (local address 127.0.98.2:41661)
0926 20:59:10.529646 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:10.529661 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:10.529735 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:10.529948 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:10.529949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:10.529951 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:10.530030 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:10.530032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.530953 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.530954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:10.532248 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:10.532250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.533122 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.533122 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:10.533222 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.533226 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:59:10.533301 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:59:10.533358 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":16}
I20260926 20:59:10.533483 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.529709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39583 (local address 127.0.98.2:36853)
0926 20:59:10.529886 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:10.529894 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:10.529972 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:10.530115 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:10.530115 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:10.530117 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:10.530174 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:10.530175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.531031 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.531032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:10.532239 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:10.532243 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.533119 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.533121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:10.533219 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.533232 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 20:59:10.533301 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:59:10.533358 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":15,"thread_start_us":94,"threads_started":1}
W20260926 20:59:10.533689 16479 tablet.cc:2369] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:59:10.534178 16702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aeb999083764dfa9232eebf0e9c6ce8" candidate_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f7e87e3a93a42589ca375b55acc1c51" is_pre_election: true
I20260926 20:59:10.534391 16702 raft_consensus.cc:2454] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a69e1ea6db94c69bc99a1e2721a3669 in term 0.
I20260926 20:59:10.534389 16432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aeb999083764dfa9232eebf0e9c6ce8" candidate_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" is_pre_election: true
I20260926 20:59:10.534533 16432 raft_consensus.cc:2454] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a69e1ea6db94c69bc99a1e2721a3669 in term 0.
I20260926 20:59:10.534626 16503 leader_election.cc:302] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a69e1ea6db94c69bc99a1e2721a3669, 8f7e87e3a93a42589ca375b55acc1c51; no voters:
I20260926 20:59:10.534838 16763 raft_consensus.cc:2781] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:59:10.534931 16763 raft_consensus.cc:482] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:59:10.534951 16763 raft_consensus.cc:3037] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:10.535694 16763 raft_consensus.cc:504] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } } peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260926 20:59:10.535851 16763 leader_election.cc:288] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [CANDIDATE]: Term 1 election: Requested vote from peers 8f7e87e3a93a42589ca375b55acc1c51 (127.0.98.3:41145), 19ed2f4f490244bf988a8f2f41dcb8f6 (127.0.98.1:39583)
I20260926 20:59:10.536048 16702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aeb999083764dfa9232eebf0e9c6ce8" candidate_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f7e87e3a93a42589ca375b55acc1c51"
I20260926 20:59:10.536080 16432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aeb999083764dfa9232eebf0e9c6ce8" candidate_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6"
I20260926 20:59:10.536147 16432 raft_consensus.cc:3037] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:10.536156 16702 raft_consensus.cc:3037] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:10.536923 16702 raft_consensus.cc:2454] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a69e1ea6db94c69bc99a1e2721a3669 in term 1.
I20260926 20:59:10.536986 16432 raft_consensus.cc:2454] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a69e1ea6db94c69bc99a1e2721a3669 in term 1.
I20260926 20:59:10.537132 16503 leader_election.cc:302] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a69e1ea6db94c69bc99a1e2721a3669, 8f7e87e3a93a42589ca375b55acc1c51; no voters:
I20260926 20:59:10.537241 16763 raft_consensus.cc:2781] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:59:10.537387 16763 raft_consensus.cc:686] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 1 LEADER]: Becoming Leader. State: Replica: 2a69e1ea6db94c69bc99a1e2721a3669, State: Running, Role: LEADER
I20260926 20:59:10.537503 16763 consensus_queue.cc:237] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } } peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260926 20:59:10.538487 16290 catalog_manager.cc:5495] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a69e1ea6db94c69bc99a1e2721a3669 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } health_report { overall_health: UNKNOWN } } }
I20260926 20:59:10.559758 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.557312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.0.98.2:34835)
0926 20:59:10.557384 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:10.557386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:10.557399 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:59:10.557445 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:10.557446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:10.557468 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:10.557544 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:59:10.557672 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.558143 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.558896 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:10.559094 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:10.559163 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:10.559309 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:59:10.559409 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:59:10.559474 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:10.559554 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 20:59:10.559574 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:59:10.561688 16432 raft_consensus.cc:1261] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Refusing update from remote peer 2a69e1ea6db94c69bc99a1e2721a3669: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:59:10.561712 16702 raft_consensus.cc:1261] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51 [term 1 FOLLOWER]: Refusing update from remote peer 2a69e1ea6db94c69bc99a1e2721a3669: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:59:10.562089 16763 consensus_queue.cc:1035] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:59:10.562248 16765 consensus_queue.cc:1035] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:59:10.565019 16770 mvcc.cc:205] Tried to move back new op lower bound from 7333709211897769984 to 7333709211802255360. Current Snapshot: MvccSnapshot[applied={T|T < 7333709211897769984}]
I20260926 20:59:10.565254 16768 mvcc.cc:205] Tried to move back new op lower bound from 7333709211897769984 to 7333709211802255360. Current Snapshot: MvccSnapshot[applied={T|T < 7333709211897769984 or (T in {7333709211897769984})}]
I20260926 20:59:10.566659 16290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:59:10.566766 16290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:59:10.567476 16290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: Sending DeleteTablet for 3 replicas of tablet 9aeb999083764dfa9232eebf0e9c6ce8
I20260926 20:59:10.567803 16550 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aeb999083764dfa9232eebf0e9c6ce8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48366
I20260926 20:59:10.567831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 16328
I20260926 20:59:10.567827 16412 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aeb999083764dfa9232eebf0e9c6ce8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60224
I20260926 20:59:10.568354 16682 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aeb999083764dfa9232eebf0e9c6ce8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60774
I20260926 20:59:10.568533 16777 tablet_replica.cc:330] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669: stopping tablet replica
I20260926 20:59:10.568642 16778 tablet_replica.cc:330] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51: stopping tablet replica
I20260926 20:59:10.568651 16777 raft_consensus.cc:2229] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:59:10.568782 16777 raft_consensus.cc:2258] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:59:10.568800 16778 raft_consensus.cc:2229] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:59:10.568996 16778 raft_consensus.cc:2258] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:59:10.569442 16777 ts_tablet_manager.cc:1905] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:59:10.569442 16778 ts_tablet_manager.cc:1905] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:59:10.571556 16778 ts_tablet_manager.cc:1918] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:59:10.571574 16777 ts_tablet_manager.cc:1918] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:59:10.571648 16777 log.cc:1196] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/9aeb999083764dfa9232eebf0e9c6ce8
I20260926 20:59:10.571753 16778 log.cc:1196] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/9aeb999083764dfa9232eebf0e9c6ce8
I20260926 20:59:10.571926 16777 ts_tablet_manager.cc:1939] T 9aeb999083764dfa9232eebf0e9c6ce8 P 2a69e1ea6db94c69bc99a1e2721a3669: Deleting consensus metadata
I20260926 20:59:10.572192 16778 ts_tablet_manager.cc:1939] T 9aeb999083764dfa9232eebf0e9c6ce8 P 8f7e87e3a93a42589ca375b55acc1c51: Deleting consensus metadata
I20260926 20:59:10.572358 16275 catalog_manager.cc:4841] TS 2a69e1ea6db94c69bc99a1e2721a3669 (127.0.98.2:34835): tablet 9aeb999083764dfa9232eebf0e9c6ce8 (table test-table [id=e579fbba16a84bc7a436b86064e6d0fb]) successfully deleted
I20260926 20:59:10.572727 16274 catalog_manager.cc:4841] TS 8f7e87e3a93a42589ca375b55acc1c51 (127.0.98.3:41145): tablet 9aeb999083764dfa9232eebf0e9c6ce8 (table test-table [id=e579fbba16a84bc7a436b86064e6d0fb]) successfully deleted
W20260926 20:59:10.574756 16274 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39583 (error 108)
I20260926 20:59:10.575196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 16484
I20260926 20:59:10.575281 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:10.575128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39583 (local address 127.0.0.1:60234)
0926 20:59:10.575198 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:10.575240 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 20:59:10.576370 16274 catalog_manager.cc:4584] TS 19ed2f4f490244bf988a8f2f41dcb8f6 (127.0.98.1:39583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9aeb999083764dfa9232eebf0e9c6ce8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39583: connect: Connection refused (error 111)
I20260926 20:59:10.580284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 16619
I20260926 20:59:10.584944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 16257
I20260926 20:59:30.590794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35223
--webserver_interface=127.0.98.62
--webserver_port=41187
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:30.674805 16780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:30.675036 16780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:30.675060 16780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:30.676368 16780 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:59:30.676445 16780 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:59:30.676460 16780 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:59:30.676472 16780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:59:30.676486 16780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:30.676501 16780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:30.676512 16780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:59:30.676525 16780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:59:30.678104 16780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:30.678324 16780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:30.678515 16780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:30.683954 16785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:30.683938 16788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:30.684036 16786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:30.684077 16780 server_base.cc:1034] running on GCE node
I20260926 20:59:30.684540 16780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:30.684854 16780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:30.686018 16780 hybrid_clock.cc:648] HybridClock initialized: now 1790456370685998 us; error 39 us; skew 500 ppm
Sep 26 20:59:30 dist-test-slave-fl7g krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:30.688979 16780 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:59:30.689529 16780 webserver.cc:460] Webserver started at http://127.0.98.62:41187/ using document root <none> and password file <none>
I20260926 20:59:30.689788 16780 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:30.689857 16780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:30.691381 16780 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:30.692317 16795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:30.692536 16780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:30.692648 16780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "a41d46a69bda4043a0b83db9c5a2fda5"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "921261bbb787c3fa6c9527d7f1498290"
server_key_iv: "907bec97c4f824e8a6d6bfac27f166eb"
server_key_version: "encryptionkey@0"
I20260926 20:59:30.693140 16780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:30.707374 16780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:30.710137 16780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:30.710282 16780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:30.714866 16780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35223
I20260926 20:59:30.714900 16847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35223 every 8 connection(s)
I20260926 20:59:30.715344 16780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:59:30.716387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 16780
I20260926 20:59:30.716763 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39583
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35223
--webserver_port=44195
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:59:30.717113 16848 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:59:30.717803 16848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: Bootstrap starting.
I20260926 20:59:30.726649 16848 log.cc:824] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:30.729153 16848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:59:30.729492 16848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: Bootstrap complete.
I20260926 20:59:30.731793 16848 raft_consensus.cc:348] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260926 20:59:30.732053 16848 raft_consensus.cc:729] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41d46a69bda4043a0b83db9c5a2fda5, State: Initialized, Role: FOLLOWER
I20260926 20:59:30.732247 16848 consensus_queue.cc:260] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260926 20:59:30.732331 16848 raft_consensus.cc:388] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:59:30.732386 16848 raft_consensus.cc:482] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:59:30.732460 16848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:59:30.733513 16848 raft_consensus.cc:504] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260926 20:59:30.733650 16848 leader_election.cc:302] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a41d46a69bda4043a0b83db9c5a2fda5; no voters:
I20260926 20:59:30.733860 16848 leader_election.cc:288] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 20:59:30.733904 16852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 20:59:30.734023 16852 raft_consensus.cc:686] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [term 3 LEADER]: Becoming Leader. State: Replica: a41d46a69bda4043a0b83db9c5a2fda5, State: Running, Role: LEADER
I20260926 20:59:30.734115 16852 consensus_queue.cc:237] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260926 20:59:30.734161 16848 sys_catalog.cc:564] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:59:30.734386 16852 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a41d46a69bda4043a0b83db9c5a2fda5. Latest consensus state: current_term: 3 leader_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } } }
I20260926 20:59:30.734467 16852 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:30.734393 16853 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d46a69bda4043a0b83db9c5a2fda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } } }
I20260926 20:59:30.734650 16853 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:30.735898 16857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:59:30.736985 16857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:59:30.737098 16857 catalog_manager.cc:1259] Loaded cluster ID: 323911ee9c1549a4ad23a6f298e828f9
I20260926 20:59:30.737130 16857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:59:30.738286 16857 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:59:30.738471 16857 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: Loaded TSK: 0
I20260926 20:59:30.738523 16857 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a41d46a69bda4043a0b83db9c5a2fda5: Loaded TSK: 1
F20260926 20:59:30.738758 16857 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe81e4e79ad google::LogMessage::Fail() at ??:0
@ 0x7fe81e4ea88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe81e4e74de google::LogMessage::Flush() at ??:0
@ 0x7fe81e4e8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe820c54104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe820c51d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe8264411cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe82644148e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe826441594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe826407c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe826411ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe8203b50d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe8203aa954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe82120e6db start_thread at ??:0
@ 0x7fe81cfe561f clone at ??:0
W20260926 20:59:30.825752 16850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:30.825968 16850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:30.825994 16850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:30.827242 16850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:30.827299 16850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:30.827330 16850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:59:30.828982 16850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44195
--tserver_master_addrs=127.0.98.62:35223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:30.829259 16850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:30.829509 16850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:30.834702 16889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:30.834896 16850 server_base.cc:1034] running on GCE node
W20260926 20:59:30.834690 16892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:30.834815 16890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:30.835197 16850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:30.835469 16850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:30.836622 16850 hybrid_clock.cc:648] HybridClock initialized: now 1790456370836603 us; error 27 us; skew 500 ppm
Sep 26 20:59:30 dist-test-slave-fl7g krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:30.839829 16850 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:59:30.840463 16850 webserver.cc:460] Webserver started at http://127.0.98.1:44195/ using document root <none> and password file <none>
I20260926 20:59:30.840770 16850 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:30.840848 16850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:30.842757 16850 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:59:30.843580 16899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:30.843782 16850 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:59:30.843883 16850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "19ed2f4f490244bf988a8f2f41dcb8f6"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "9f5c8299948bad12ea10cc2044e870f4"
server_key_iv: "d14a3101e2e9e4cfe69e0db859c1a2ca"
server_key_version: "encryptionkey@0"
I20260926 20:59:30.844377 16850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:30.879561 16850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:30.882575 16850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:30.882756 16850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:30.883023 16850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:30.883550 16906 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:59:30.885260 16850 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:59:30.885332 16850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 20:59:30.885370 16850 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:59:30.886303 16850 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:59:30.886353 16850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 20:59:30.886462 16906 tablet_bootstrap.cc:492] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6: Bootstrap starting.
I20260926 20:59:30.893594 16850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39583
I20260926 20:59:30.893859 17013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39583 every 8 connection(s)
I20260926 20:59:30.894129 16850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:59:30.897652 16906 log.cc:824] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:30.899919 16906 tablet_bootstrap.cc:492] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:59:30.900231 16906 tablet_bootstrap.cc:492] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6: Bootstrap complete.
I20260926 20:59:30.901208 16906 ts_tablet_manager.cc:1397] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20260926 20:59:30.902323 16906 raft_consensus.cc:348] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } } peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260926 20:59:30.902580 16906 raft_consensus.cc:729] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ed2f4f490244bf988a8f2f41dcb8f6, State: Initialized, Role: FOLLOWER
I20260926 20:59:30.902769 16906 consensus_queue.cc:260] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } } peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260926 20:59:30.902880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 16850
I20260926 20:59:30.903100 16906 ts_tablet_manager.cc:1428] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 20:59:30.903174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34835
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35223
--webserver_port=46443
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:30.957578 17016 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:59:30.894919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:36511)
0926 20:59:30.895151 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:30.895211 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:30.895871 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:30.957359 (+ 61488us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":102,"threads_started":1}
W20260926 20:59:30.957880 17014 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35223 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35223: Client connection negotiation failed: client connection to 127.0.98.62:35223: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:59:30.958263 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:30.958138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:56929)
0926 20:59:30.958194 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:30.958241 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 20:59:30.958653 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:30.958540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:43873)
0926 20:59:30.958592 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:30.958624 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 20:59:30.958830 17014 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:59:30.993053 17019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:30.993330 17019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:30.993361 17019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:30.994594 17019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:30.994642 17019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:30.994670 17019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:59:30.996155 17019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46443
--tserver_master_addrs=127.0.98.62:35223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:30.996457 17019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:30.996738 17019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:31.002002 17027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:31.002002 17024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:31.002017 17019 server_base.cc:1034] running on GCE node
W20260926 20:59:31.002099 17025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:31.002426 17019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:31.002687 17019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:31.003878 17019 hybrid_clock.cc:648] HybridClock initialized: now 1790456371003870 us; error 36 us; skew 500 ppm
Sep 26 20:59:31 dist-test-slave-fl7g krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:31.006441 17019 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:59:31.006928 17019 webserver.cc:460] Webserver started at http://127.0.98.2:46443/ using document root <none> and password file <none>
I20260926 20:59:31.007117 17019 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:31.007165 17019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:31.008635 17019 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:59:31.009361 17034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:31.009567 17019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:31.009654 17019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "2a69e1ea6db94c69bc99a1e2721a3669"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "fa55bd102089ba53e48c9fc27dfb2037"
server_key_iv: "cfd08c987b6a90e5aa90057405c0a14d"
server_key_version: "encryptionkey@0"
I20260926 20:59:31.010035 17019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:31.024334 17019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:31.026430 17019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:31.026554 17019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:31.026765 17019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:31.027110 17019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:31.027155 17019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:31.027201 17019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:31.027228 17019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:31.033635 17019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34835
I20260926 20:59:31.033722 17147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34835 every 8 connection(s)
I20260926 20:59:31.034044 17019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:59:31.035214 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:31.034794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:34387)
0926 20:59:31.034977 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:31.035058 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":77,"threads_started":1}
W20260926 20:59:31.035753 17148 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35223 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35223: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
I20260926 20:59:31.036113 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:31.036006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:37779)
0926 20:59:31.036056 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:31.036089 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 20:59:31.036496 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:31.036379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:34075)
0926 20:59:31.036451 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:31.036475 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 20:59:31.036609 17148 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:59:31.038827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 17019
I20260926 20:59:31.039098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41145
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35223
--webserver_port=42219
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:31.120478 17151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:31.120708 17151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:31.120736 17151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:31.121965 17151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:31.122012 17151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:31.122040 17151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:59:31.123528 17151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42219
--tserver_master_addrs=127.0.98.62:35223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:31.123788 17151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:31.124033 17151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:31.129369 17157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:31.129400 17151 server_base.cc:1034] running on GCE node
W20260926 20:59:31.129379 17156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:31.129380 17159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:31.129940 17151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:31.130224 17151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:31.131402 17151 hybrid_clock.cc:648] HybridClock initialized: now 1790456371131386 us; error 28 us; skew 500 ppm
Sep 26 20:59:31 dist-test-slave-fl7g krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:31.134182 17151 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:59:31.134688 17151 webserver.cc:460] Webserver started at http://127.0.98.3:42219/ using document root <none> and password file <none>
I20260926 20:59:31.134913 17151 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:31.134979 17151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:31.136385 17151 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:31.137041 17166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:31.137213 17151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:31.137312 17151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "8f7e87e3a93a42589ca375b55acc1c51"
format_stamp: "Formatted at 2026-09-26 20:58:49 on dist-test-slave-fl7g"
server_key: "dd257e2166d2996642375f4366bf90b5"
server_key_iv: "2fafac7838d0c695abcc23da0bc5faed"
server_key_version: "encryptionkey@0"
I20260926 20:59:31.137676 17151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:31.148950 17151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:31.151526 17151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:31.151743 17151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:31.152014 17151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:31.152345 17151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:31.152379 17151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:31.152464 17151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:31.152494 17151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:31.158459 17151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41145
I20260926 20:59:31.158509 17279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41145 every 8 connection(s)
I20260926 20:59:31.158859 17151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:59:31.159921 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:31.159550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:37805)
0926 20:59:31.159708 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:31.159782 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
W20260926 20:59:31.160341 17280 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35223 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35223: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
I20260926 20:59:31.160668 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:31.160565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:53873)
0926 20:59:31.160622 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:31.160649 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 20:59:31.160923 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:31.160867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:37673)
0926 20:59:31.160891 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:31.160910 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 20:59:31.161047 17280 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:59:31.164063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 17151
I20260926 20:59:31.959548 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:31.959268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:46977)
0926 20:59:31.959426 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:31.959479 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260926 20:59:32.037228 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:32.036976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:53505)
0926 20:59:32.037114 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:32.037171 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260926 20:59:32.161597 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:32.161389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:33225)
0926 20:59:32.161501 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:32.161543 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
I20260926 20:59:32.418038 17287 raft_consensus.cc:482] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:59:32.418179 17287 raft_consensus.cc:504] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } } peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260926 20:59:32.418476 17287 leader_election.cc:288] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f7e87e3a93a42589ca375b55acc1c51 (127.0.98.3:41145), 2a69e1ea6db94c69bc99a1e2721a3669 (127.0.98.2:34835)
Sep 26 20:59:32 dist-test-slave-fl7g krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 26 20:59:32 dist-test-slave-fl7g krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 20:59:32.430536 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:32.418697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41145 (local address 127.0.98.1:59505)
0926 20:59:32.418796 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:32.418809 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:32.418873 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:32.419720 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:32.419723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:32.420224 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:32.420557 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:32.420560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:32.421839 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:32.421840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:32.422635 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:32.422638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:32.423296 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:32.423297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:32.423394 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:32.424043 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:59:32.424048 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:32.425711 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:32.428232 (+ 2521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:32.428244 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:32.428247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:32.428856 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:32.429184 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:32.429185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:32.429186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:32.429371 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:32.429864 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:32.429871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:32.430043 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 20:59:32.430179 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 20:59:32.430388 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":31}
I20260926 20:59:32.430585 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:32.418799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59505 (local address 127.0.98.3:41145)
0926 20:59:32.418976 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:32.418988 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:32.419024 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:59:32.419477 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:32.419479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:32.419525 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:32.419682 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:32.420695 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:32.421576 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:32.422741 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:32.423174 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:32.423314 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:32.425880 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:32.425899 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:32.425899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:32.425917 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:32.428119 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:32.428952 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:32.428956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:32.428956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:32.429101 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:32.429450 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:32.429451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:32.429452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:32.429798 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:32.429877 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:32.430309 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 20:59:32.430475 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":55,"threads_started":1}
I20260926 20:59:32.430585 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:32.418855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50687 (local address 127.0.98.2:34835)
0926 20:59:32.419025 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:32.419033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:32.419057 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:59:32.419491 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:32.419492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:32.419530 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:32.419678 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:32.420672 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:32.421576 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:32.422764 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:32.423174 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:32.423300 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:32.426211 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:32.426215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:32.426216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:32.426230 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:32.428091 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:32.428952 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:32.428956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:32.428956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:32.429101 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:32.429450 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:32.429451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:32.429452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:32.429790 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:32.429877 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:32.430309 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 20:59:32.430462 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260926 20:59:32.430548 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:32.418697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34835 (local address 127.0.98.1:50687)
0926 20:59:32.418826 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:32.418839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:32.418914 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:32.419747 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:32.419748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:32.420325 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:32.420547 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:32.420556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:32.421800 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:32.421802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:32.422636 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:32.422639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:32.423271 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:32.423272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:32.423394 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:32.423986 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:59:32.423995 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:32.426091 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:32.428232 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:32.428244 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:32.428247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:32.428856 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:32.429185 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:32.429185 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:32.429186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:32.429371 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:32.429864 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:32.429871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:32.430043 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 20:59:32.430173 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 20:59:32.430447 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":212,"thread_start_us":56,"threads_started":1}
W20260926 20:59:32.431624 16900 leader_election.cc:341] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f7e87e3a93a42589ca375b55acc1c51 (127.0.98.3:41145): Not found: Tablet not found: 9aeb999083764dfa9232eebf0e9c6ce8
W20260926 20:59:32.432003 16901 leader_election.cc:341] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a69e1ea6db94c69bc99a1e2721a3669 (127.0.98.2:34835): Not found: Tablet not found: 9aeb999083764dfa9232eebf0e9c6ce8
I20260926 20:59:32.432094 16901 leader_election.cc:302] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19ed2f4f490244bf988a8f2f41dcb8f6; no voters: 2a69e1ea6db94c69bc99a1e2721a3669, 8f7e87e3a93a42589ca375b55acc1c51
I20260926 20:59:32.432242 17287 raft_consensus.cc:2726] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:59:32.960510 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:32.960239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:50293)
0926 20:59:32.960370 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:32.960451 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260926 20:59:33.038064 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:33.037799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:37259)
0926 20:59:33.037957 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:33.038015 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
I20260926 20:59:33.162436 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:33.162193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:56685)
0926 20:59:33.162340 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:33.162389 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
I20260926 20:59:33.961326 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:33.961069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:42221)
0926 20:59:33.961228 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:33.961277 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260926 20:59:34.038902 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:34.038638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:38711)
0926 20:59:34.038809 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:34.038855 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":74,"threads_started":1}
I20260926 20:59:34.163316 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:34.163042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:49251)
0926 20:59:34.163223 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:34.163270 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
I20260926 20:59:34.488312 17297 raft_consensus.cc:482] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:59:34.488428 17297 raft_consensus.cc:504] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } } peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260926 20:59:34.488605 17297 leader_election.cc:288] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f7e87e3a93a42589ca375b55acc1c51 (127.0.98.3:41145), 2a69e1ea6db94c69bc99a1e2721a3669 (127.0.98.2:34835)
W20260926 20:59:34.489032 16901 leader_election.cc:341] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a69e1ea6db94c69bc99a1e2721a3669 (127.0.98.2:34835): Not found: Tablet not found: 9aeb999083764dfa9232eebf0e9c6ce8
W20260926 20:59:34.489130 16900 leader_election.cc:341] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f7e87e3a93a42589ca375b55acc1c51 (127.0.98.3:41145): Not found: Tablet not found: 9aeb999083764dfa9232eebf0e9c6ce8
I20260926 20:59:34.489214 16900 leader_election.cc:302] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19ed2f4f490244bf988a8f2f41dcb8f6; no voters: 2a69e1ea6db94c69bc99a1e2721a3669, 8f7e87e3a93a42589ca375b55acc1c51
I20260926 20:59:34.489315 17297 raft_consensus.cc:2726] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:59:34.962184 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:34.961878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:55573)
0926 20:59:34.962061 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:34.962131 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":77,"threads_started":1}
I20260926 20:59:35.039758 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:35.039513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:40305)
0926 20:59:35.039661 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:35.039709 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260926 20:59:35.164214 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:35.163990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:48489)
0926 20:59:35.164120 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:35.164165 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260926 20:59:35.963054 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:35.962825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:54955)
0926 20:59:35.962961 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:35.963007 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260926 20:59:36.040591 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:36.040307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:53333)
0926 20:59:36.040492 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:36.040545 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":79,"threads_started":1}
I20260926 20:59:36.165138 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:36.164883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:50287)
0926 20:59:36.165041 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:36.165091 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
I20260926 20:59:36.359043 17304 raft_consensus.cc:482] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:59:36.359150 17304 raft_consensus.cc:504] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } } peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260926 20:59:36.359351 17304 leader_election.cc:288] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f7e87e3a93a42589ca375b55acc1c51 (127.0.98.3:41145), 2a69e1ea6db94c69bc99a1e2721a3669 (127.0.98.2:34835)
W20260926 20:59:36.359783 16900 leader_election.cc:341] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f7e87e3a93a42589ca375b55acc1c51 (127.0.98.3:41145): Not found: Tablet not found: 9aeb999083764dfa9232eebf0e9c6ce8
W20260926 20:59:36.359910 16901 leader_election.cc:341] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a69e1ea6db94c69bc99a1e2721a3669 (127.0.98.2:34835): Not found: Tablet not found: 9aeb999083764dfa9232eebf0e9c6ce8
I20260926 20:59:36.359982 16901 leader_election.cc:302] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19ed2f4f490244bf988a8f2f41dcb8f6; no voters: 2a69e1ea6db94c69bc99a1e2721a3669, 8f7e87e3a93a42589ca375b55acc1c51
I20260926 20:59:36.360061 17304 raft_consensus.cc:2726] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:59:36.964001 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:36.963729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:46305)
0926 20:59:36.963899 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:36.963946 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":90,"threads_started":1}
I20260926 20:59:37.041489 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:37.041226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:38101)
0926 20:59:37.041395 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:37.041442 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":80,"threads_started":1}
I20260926 20:59:37.166072 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:37.165795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:39557)
0926 20:59:37.165975 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:37.166024 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260926 20:59:37.964960 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:37.964710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:47297)
0926 20:59:37.964861 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:37.964913 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
I20260926 20:59:38.042254 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:38.042038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:37303)
0926 20:59:38.042162 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:38.042206 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260926 20:59:38.166885 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:38.166652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:38715)
0926 20:59:38.166792 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:38.166839 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260926 20:59:38.965850 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:38.965564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:45883)
0926 20:59:38.965754 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:38.965802 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":112,"threads_started":1}
I20260926 20:59:39.042944 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:39.042716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:54221)
0926 20:59:39.042866 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:39.042896 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260926 20:59:39.167740 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:39.167496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:47339)
0926 20:59:39.167637 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:39.167685 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260926 20:59:39.842383 17314 raft_consensus.cc:482] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:59:39.842499 17314 raft_consensus.cc:504] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } } peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260926 20:59:39.842772 17314 leader_election.cc:288] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f7e87e3a93a42589ca375b55acc1c51 (127.0.98.3:41145), 2a69e1ea6db94c69bc99a1e2721a3669 (127.0.98.2:34835)
W20260926 20:59:39.843192 16900 leader_election.cc:341] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f7e87e3a93a42589ca375b55acc1c51 (127.0.98.3:41145): Not found: Tablet not found: 9aeb999083764dfa9232eebf0e9c6ce8
W20260926 20:59:39.843323 16901 leader_election.cc:341] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a69e1ea6db94c69bc99a1e2721a3669 (127.0.98.2:34835): Not found: Tablet not found: 9aeb999083764dfa9232eebf0e9c6ce8
I20260926 20:59:39.843381 16901 leader_election.cc:302] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19ed2f4f490244bf988a8f2f41dcb8f6; no voters: 2a69e1ea6db94c69bc99a1e2721a3669, 8f7e87e3a93a42589ca375b55acc1c51
I20260926 20:59:39.843469 17314 raft_consensus.cc:2726] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:59:39.966725 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:39.966475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:43419)
0926 20:59:39.966631 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:39.966678 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260926 20:59:40.043797 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:40.043452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:38163)
0926 20:59:40.043682 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:40.043725 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":64,"threads_started":1}
I20260926 20:59:40.168733 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:40.168394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:53033)
0926 20:59:40.168604 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:40.168651 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":70,"threads_started":1}
I20260926 20:59:40.967576 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:40.967324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:39617)
0926 20:59:40.967487 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:40.967528 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":68,"threads_started":1}
I20260926 20:59:41.044610 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:41.044310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:60931)
0926 20:59:41.044516 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:41.044562 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":64,"threads_started":1}
I20260926 20:59:41.169593 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:41.169329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:55717)
0926 20:59:41.169495 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:41.169543 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":72,"threads_started":1}
I20260926 20:59:41.968398 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:41.968172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:58237)
0926 20:59:41.968310 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:41.968351 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260926 20:59:42.045483 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:42.045215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:39031)
0926 20:59:42.045386 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:42.045433 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260926 20:59:42.170523 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:42.170260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:49717)
0926 20:59:42.170419 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:42.170474 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260926 20:59:42.969175 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:42.968945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:58005)
0926 20:59:42.969091 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:42.969126 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260926 20:59:43.046336 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:43.046086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:40505)
0926 20:59:43.046228 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:43.046272 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260926 20:59:43.171262 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:43.171012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:34409)
0926 20:59:43.171168 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:43.171208 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":63,"threads_started":1}
I20260926 20:59:43.969967 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:43.969710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:41423)
0926 20:59:43.969872 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:43.969919 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":66,"threads_started":1}
I20260926 20:59:44.047211 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:44.046925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:45685)
0926 20:59:44.047113 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:44.047158 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":82,"threads_started":1}
I20260926 20:59:44.172233 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:44.171934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:49989)
0926 20:59:44.172100 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:44.172148 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260926 20:59:44.970732 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:44.970503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:53671)
0926 20:59:44.970646 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:44.970685 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260926 20:59:45.048059 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:45.047812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:49091)
0926 20:59:45.047962 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:45.048012 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260926 20:59:45.173171 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:45.172877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:47079)
0926 20:59:45.173065 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:45.173113 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":98,"threads_started":1}
I20260926 20:59:45.390254 17333 raft_consensus.cc:482] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:59:45.390419 17333 raft_consensus.cc:504] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f7e87e3a93a42589ca375b55acc1c51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41145 } } peers { permanent_uuid: "2a69e1ea6db94c69bc99a1e2721a3669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34835 } } peers { permanent_uuid: "19ed2f4f490244bf988a8f2f41dcb8f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260926 20:59:45.390654 17333 leader_election.cc:288] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f7e87e3a93a42589ca375b55acc1c51 (127.0.98.3:41145), 2a69e1ea6db94c69bc99a1e2721a3669 (127.0.98.2:34835)
W20260926 20:59:45.391080 16901 leader_election.cc:341] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a69e1ea6db94c69bc99a1e2721a3669 (127.0.98.2:34835): Not found: Tablet not found: 9aeb999083764dfa9232eebf0e9c6ce8
W20260926 20:59:45.391163 16900 leader_election.cc:341] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8f7e87e3a93a42589ca375b55acc1c51 (127.0.98.3:41145): Not found: Tablet not found: 9aeb999083764dfa9232eebf0e9c6ce8
I20260926 20:59:45.391250 16900 leader_election.cc:302] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19ed2f4f490244bf988a8f2f41dcb8f6; no voters: 2a69e1ea6db94c69bc99a1e2721a3669, 8f7e87e3a93a42589ca375b55acc1c51
I20260926 20:59:45.391342 17333 raft_consensus.cc:2726] T 9aeb999083764dfa9232eebf0e9c6ce8 P 19ed2f4f490244bf988a8f2f41dcb8f6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:59:45.971575 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:45.971337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:49619)
0926 20:59:45.971484 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:45.971528 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260926 20:59:46.049240 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:46.048920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:57593)
0926 20:59:46.049109 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:46.049185 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
I20260926 20:59:46.170809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 16850
I20260926 20:59:46.174057 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:46.173787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:42541)
0926 20:59:46.173942 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:46.173994 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":57,"threads_started":1}
I20260926 20:59:46.175911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 17019
I20260926 20:59:46.179981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 17151
2026-09-26T20:59:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58410 ms)
[----------] 30 tests from SecurityITest (235390 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:59:46 dist-test-slave-fl7g krb5kdc[17339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:59:46 dist-test-slave-fl7g krb5kdc[17339](info): set up 2 sockets
Sep 26 20:59:46 dist-test-slave-fl7g krb5kdc[17339](info): commencing operation
krb5kdc: starting...
W20260926 20:59:47.232168 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:59:47 dist-test-slave-fl7g krb5kdc[17339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:59:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:59:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:59:47.355454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39781
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:47.438551 17352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:47.438825 17352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:47.438866 17352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:47.440058 17352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:59:47.440140 17352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:47.440167 17352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:47.440198 17352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:59:47.440230 17352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:59:47.441823 17352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:47.442106 17352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:47.442354 17352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:47.447570 17357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:47.447584 17358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:47.447584 17360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:47.447654 17352 server_base.cc:1034] running on GCE node
I20260926 20:59:47.448056 17352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:47.448369 17352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:47.449550 17352 hybrid_clock.cc:648] HybridClock initialized: now 1790456387449529 us; error 39 us; skew 500 ppm
Sep 26 20:59:47 dist-test-slave-fl7g krb5kdc[17339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:47.452131 17352 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:59:47.452613 17352 webserver.cc:460] Webserver started at http://127.0.98.62:38097/ using document root <none> and password file <none>
I20260926 20:59:47.452822 17352 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:47.452863 17352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:47.452951 17352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:47.453975 17352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "27a783c3259546c4b98376e17bf1adfd"
format_stamp: "Formatted at 2026-09-26 20:59:47 on dist-test-slave-fl7g"
server_key: "2a523b4f4ac0bc68a617fd44c1cf6ccc"
server_key_iv: "3010cf29fcb92122b340fcbf6ad2dae4"
server_key_version: "encryptionkey@0"
I20260926 20:59:47.454277 17352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "27a783c3259546c4b98376e17bf1adfd"
format_stamp: "Formatted at 2026-09-26 20:59:47 on dist-test-slave-fl7g"
server_key: "2a523b4f4ac0bc68a617fd44c1cf6ccc"
server_key_iv: "3010cf29fcb92122b340fcbf6ad2dae4"
server_key_version: "encryptionkey@0"
I20260926 20:59:47.455552 17352 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:59:47.456226 17367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:47.456480 17352 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:47.456552 17352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "27a783c3259546c4b98376e17bf1adfd"
format_stamp: "Formatted at 2026-09-26 20:59:47 on dist-test-slave-fl7g"
server_key: "2a523b4f4ac0bc68a617fd44c1cf6ccc"
server_key_iv: "3010cf29fcb92122b340fcbf6ad2dae4"
server_key_version: "encryptionkey@0"
I20260926 20:59:47.456606 17352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:47.482178 17352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:47.484436 17352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:47.484549 17352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:47.488252 17352 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39781
I20260926 20:59:47.488330 17419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39781 every 8 connection(s)
I20260926 20:59:47.488703 17352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:59:47.489617 17420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:47.490762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 17352
I20260926 20:59:47.490892 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:59:47.491089 392 external_mini_cluster.cc:1428] Setting key 0078116560ea96428c3dd76eebe546e6
I20260926 20:59:47.492496 17420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd: Bootstrap starting.
I20260926 20:59:47.493131 17420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:47.493450 17420 log.cc:824] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:47.494163 17420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd: No bootstrap required, opened a new log
I20260926 20:59:47.496125 17420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27a783c3259546c4b98376e17bf1adfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39781 } }
I20260926 20:59:47.496309 17420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:47.496374 17420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a783c3259546c4b98376e17bf1adfd, State: Initialized, Role: FOLLOWER
I20260926 20:59:47.496537 17420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27a783c3259546c4b98376e17bf1adfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39781 } }
I20260926 20:59:47.496624 17420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:59:47.496670 17420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:59:47.496716 17420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:47.497310 17420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27a783c3259546c4b98376e17bf1adfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39781 } }
I20260926 20:59:47.497424 17420 leader_election.cc:302] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27a783c3259546c4b98376e17bf1adfd; no voters:
I20260926 20:59:47.497592 17420 leader_election.cc:288] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:59:47.497654 17425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:59:47.497829 17425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd [term 1 LEADER]: Becoming Leader. State: Replica: 27a783c3259546c4b98376e17bf1adfd, State: Running, Role: LEADER
Sep 26 20:59:47 dist-test-slave-fl7g krb5kdc[17339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:59:47.497979 17425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27a783c3259546c4b98376e17bf1adfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39781 } }
I20260926 20:59:47.497861 17420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:59:47.499080 17427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27a783c3259546c4b98376e17bf1adfd. Latest consensus state: current_term: 1 leader_uuid: "27a783c3259546c4b98376e17bf1adfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27a783c3259546c4b98376e17bf1adfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39781 } } }
I20260926 20:59:47.499173 17427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:47.499485 17426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27a783c3259546c4b98376e17bf1adfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27a783c3259546c4b98376e17bf1adfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39781 } } }
I20260926 20:59:47.499611 17426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:47.499627 17435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:59:47.500356 17435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:59:47.502286 17435 catalog_manager.cc:1347] Generated new cluster ID: d4335f54b85f4c17a2c37ebc8ef91119
I20260926 20:59:47.502364 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:47.491628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55912 (local address 127.0.98.62:39781)
0926 20:59:47.491818 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:47.491828 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:47.491866 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:59:47.492434 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:47.492443 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:47.492506 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:47.492656 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:47.493264 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:47.494215 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:47.495033 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:47.495453 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:47.495573 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:47.498430 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:47.498436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:47.498436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:47.498451 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:47.500638 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:47.501033 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:47.501037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:47.501037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:47.501145 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:47.501384 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:47.501385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:47.501385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:47.501693 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:47.501851 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:47.502005 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 20:59:47.502157 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
I20260926 20:59:47.502381 17435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:59:47.509840 17435 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:59:47.510403 17435 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:59:47.517553 17435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd: Generated new TSK 0
I20260926 20:59:47.517846 17435 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:59:47.549710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39781
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:47.640825 17446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:47.641049 17446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:47.641077 17446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:47.642361 17446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:47.642416 17446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:47.642447 17446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:59:47.644209 17446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:47.644505 17446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:47.644757 17446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:47.650761 17452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:47.650894 17454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:47.651003 17446 server_base.cc:1034] running on GCE node
W20260926 20:59:47.651046 17451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:47.651275 17446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:47.651517 17446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:47.652704 17446 hybrid_clock.cc:648] HybridClock initialized: now 1790456387652690 us; error 44 us; skew 500 ppm
Sep 26 20:59:47 dist-test-slave-fl7g krb5kdc[17339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:47.655139 17446 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:59:47.655637 17446 webserver.cc:460] Webserver started at http://127.0.98.1:36619/ using document root <none> and password file <none>
I20260926 20:59:47.655889 17446 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:47.655956 17446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:47.656101 17446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:47.657241 17446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "4cd7b9a40cbc43698a0ec388a5d6c552"
format_stamp: "Formatted at 2026-09-26 20:59:47 on dist-test-slave-fl7g"
server_key: "383a9f508107349869b7c7cafcc3a268"
server_key_iv: "3959baa789f4a2269d98eae6ceed0604"
server_key_version: "encryptionkey@0"
I20260926 20:59:47.657598 17446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4cd7b9a40cbc43698a0ec388a5d6c552"
format_stamp: "Formatted at 2026-09-26 20:59:47 on dist-test-slave-fl7g"
server_key: "383a9f508107349869b7c7cafcc3a268"
server_key_iv: "3959baa789f4a2269d98eae6ceed0604"
server_key_version: "encryptionkey@0"
I20260926 20:59:47.658939 17446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:59:47.659788 17461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:47.659986 17446 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:47.660106 17446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "4cd7b9a40cbc43698a0ec388a5d6c552"
format_stamp: "Formatted at 2026-09-26 20:59:47 on dist-test-slave-fl7g"
server_key: "383a9f508107349869b7c7cafcc3a268"
server_key_iv: "3959baa789f4a2269d98eae6ceed0604"
server_key_version: "encryptionkey@0"
I20260926 20:59:47.660187 17446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:47.679378 17446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:47.681677 17446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:47.681871 17446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:47.682119 17446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:47.682452 17446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:47.682508 17446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:47.682552 17446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:47.682580 17446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:47.688684 17446 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37499
I20260926 20:59:47.688747 17574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37499 every 8 connection(s)
I20260926 20:59:47.689086 17446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 20:59:47 dist-test-slave-fl7g krb5kdc[17339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:59:47.695811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 17446
I20260926 20:59:47.695922 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:59:47.696177 392 external_mini_cluster.cc:1428] Setting key 1210b57aab2d1eb2439dede0d6e98842
I20260926 20:59:47.699633 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:47.689833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48033 (local address 127.0.98.62:39781)
0926 20:59:47.689878 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:47.689880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:47.690133 (+ 253us) server_negotiation.cc:407] Connection header received
0926 20:59:47.691127 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:47.691129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:47.691141 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:47.691195 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:47.691862 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:47.692388 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:47.693458 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:47.693685 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:47.693772 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:47.696132 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:47.696133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:47.696134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:47.696141 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:47.697832 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:47.698409 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:47.698410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:47.698410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:47.698482 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:47.698741 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:47.698742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:47.698743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:47.698929 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:47.699027 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:47.699352 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 20:59:47.699531 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:59:47.699807 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:47.689803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39781 (local address 127.0.98.1:48033)
0926 20:59:47.689950 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:47.690019 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:47.691005 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:47.691283 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:47.691292 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:47.691563 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:47.691758 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:47.691762 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:47.692516 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:47.692517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:47.693361 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:47.693363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:47.693785 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:47.693786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:47.693924 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:47.694445 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:59:47.694453 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:47.696021 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:47.697920 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:47.697929 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:47.697931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:47.698292 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:47.698551 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:47.698552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:47.698552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:47.698659 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:47.699015 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:47.699022 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:47.699193 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 20:59:47.699314 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 20:59:47.699505 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260926 20:59:47.700556 17575 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39781
I20260926 20:59:47.700688 17575 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:47.700891 17575 heartbeater.cc:511] Master 127.0.98.62:39781 requested a full tablet report, sending...
I20260926 20:59:47.701537 17384 ts_manager.cc:194] Registered new tserver with Master: 4cd7b9a40cbc43698a0ec388a5d6c552 (127.0.98.1:37499)
I20260926 20:59:47.702585 17384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48033
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:59:47.723886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39781
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:47.809273 17580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:47.809513 17580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:47.809538 17580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:47.810729 17580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:47.810770 17580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:47.810798 17580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:59:47.812354 17580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:47.812659 17580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:47.812861 17580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:47.818214 17588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:47.818187 17585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:47.818317 17580 server_base.cc:1034] running on GCE node
W20260926 20:59:47.818176 17586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:47.818789 17580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:47.819095 17580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:47.820289 17580 hybrid_clock.cc:648] HybridClock initialized: now 1790456387820283 us; error 54 us; skew 500 ppm
Sep 26 20:59:47 dist-test-slave-fl7g krb5kdc[17339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:47.823064 17580 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:59:47.823477 17580 webserver.cc:460] Webserver started at http://127.0.98.2:43927/ using document root <none> and password file <none>
I20260926 20:59:47.823733 17580 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:47.823782 17580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:47.823946 17580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:47.825125 17580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "a236d196459c4a98a7cf3a7a418e874b"
format_stamp: "Formatted at 2026-09-26 20:59:47 on dist-test-slave-fl7g"
server_key: "ed12dd5ff5b01ca64dfccdece043ad8b"
server_key_iv: "369df85c1667124caffae1b79960130e"
server_key_version: "encryptionkey@0"
I20260926 20:59:47.825500 17580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a236d196459c4a98a7cf3a7a418e874b"
format_stamp: "Formatted at 2026-09-26 20:59:47 on dist-test-slave-fl7g"
server_key: "ed12dd5ff5b01ca64dfccdece043ad8b"
server_key_iv: "369df85c1667124caffae1b79960130e"
server_key_version: "encryptionkey@0"
I20260926 20:59:47.826897 17580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:59:47.827718 17595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:47.827913 17580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:47.828020 17580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "a236d196459c4a98a7cf3a7a418e874b"
format_stamp: "Formatted at 2026-09-26 20:59:47 on dist-test-slave-fl7g"
server_key: "ed12dd5ff5b01ca64dfccdece043ad8b"
server_key_iv: "369df85c1667124caffae1b79960130e"
server_key_version: "encryptionkey@0"
I20260926 20:59:47.828099 17580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:47.844766 17580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:47.847280 17580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:47.847456 17580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:47.847716 17580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:47.848042 17580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:47.848096 17580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:47.848140 17580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:47.848169 17580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:47.854090 17580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43035
I20260926 20:59:47.854220 17708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43035 every 8 connection(s)
I20260926 20:59:47.854506 17580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:59:47.859383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 17580
I20260926 20:59:47.859509 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:59:47.859740 392 external_mini_cluster.cc:1428] Setting key c738f775df9a368c67d6e7c6ca6987a1
Sep 26 20:59:47 dist-test-slave-fl7g krb5kdc[17339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:59:47.865464 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:47.855320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39781 (local address 127.0.98.2:48713)
0926 20:59:47.855490 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:47.855526 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:47.856642 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:47.856904 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:47.856907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:47.857174 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:47.857379 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:47.857384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:47.858044 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:47.858045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:47.858879 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:47.858881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:47.859342 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:47.859343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:47.859511 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:47.860228 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:59:47.860256 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:47.861898 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:47.863956 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:47.863962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:47.863964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:47.864233 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:47.864470 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:47.864470 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:47.864471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:47.864591 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:47.864923 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:47.864932 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:47.865063 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 20:59:47.865154 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 20:59:47.865291 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":107,"threads_started":1}
I20260926 20:59:47.865455 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:47.855360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48713 (local address 127.0.98.62:39781)
0926 20:59:47.855411 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:47.855413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:47.855622 (+ 209us) server_negotiation.cc:407] Connection header received
0926 20:59:47.856767 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:47.856769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:47.856781 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:47.856843 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:47.857452 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:47.857961 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:47.858975 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:47.859246 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:47.859317 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:47.862009 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:47.862011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:47.862012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:47.862020 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:47.863875 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:47.864315 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:47.864316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:47.864316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:47.864387 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:47.864662 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:47.864664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:47.864664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:47.864836 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:47.864917 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:47.865218 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 20:59:47.865314 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:59:47.866055 17709 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39781
I20260926 20:59:47.866181 17709 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:47.866391 17709 heartbeater.cc:511] Master 127.0.98.62:39781 requested a full tablet report, sending...
I20260926 20:59:47.866873 17384 ts_manager.cc:194] Registered new tserver with Master: a236d196459c4a98a7cf3a7a418e874b (127.0.98.2:43035)
I20260926 20:59:47.867419 17384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48713
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:59:47.888356 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39781
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:47.972246 17714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:47.972561 17714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:47.972616 17714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:47.973810 17714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:47.973882 17714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:47.973929 17714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:59:47.975450 17714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:47.975725 17714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:47.975953 17714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:47.981120 17722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:47.981113 17720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:47.981113 17719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:47.981477 17714 server_base.cc:1034] running on GCE node
I20260926 20:59:47.981729 17714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:47.981981 17714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:47.983143 17714 hybrid_clock.cc:648] HybridClock initialized: now 1790456387983139 us; error 38 us; skew 500 ppm
Sep 26 20:59:47 dist-test-slave-fl7g krb5kdc[17339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:47.985764 17714 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:59:47.986250 17714 webserver.cc:460] Webserver started at http://127.0.98.3:37215/ using document root <none> and password file <none>
I20260926 20:59:47.986461 17714 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:47.986505 17714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:47.986596 17714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:47.987746 17714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "1a2f009db6ba4d6195f35435b95d0b62"
format_stamp: "Formatted at 2026-09-26 20:59:47 on dist-test-slave-fl7g"
server_key: "3b7600c0430ea82c83e7d9ccd0f39592"
server_key_iv: "11fcf749b9145439e6c1e60d3d78c136"
server_key_version: "encryptionkey@0"
I20260926 20:59:47.988052 17714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1a2f009db6ba4d6195f35435b95d0b62"
format_stamp: "Formatted at 2026-09-26 20:59:47 on dist-test-slave-fl7g"
server_key: "3b7600c0430ea82c83e7d9ccd0f39592"
server_key_iv: "11fcf749b9145439e6c1e60d3d78c136"
server_key_version: "encryptionkey@0"
I20260926 20:59:47.989457 17714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:59:47.990275 17729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:47.990454 17714 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:59:47.990527 17714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "1a2f009db6ba4d6195f35435b95d0b62"
format_stamp: "Formatted at 2026-09-26 20:59:47 on dist-test-slave-fl7g"
server_key: "3b7600c0430ea82c83e7d9ccd0f39592"
server_key_iv: "11fcf749b9145439e6c1e60d3d78c136"
server_key_version: "encryptionkey@0"
I20260926 20:59:47.990653 17714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:48.014880 17714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:48.017540 17714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:48.017735 17714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:48.017946 17714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:48.018267 17714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:48.018324 17714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:48.018370 17714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:48.018394 17714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:48.024401 17714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34657
I20260926 20:59:48.024477 17842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34657 every 8 connection(s)
I20260926 20:59:48.024801 17714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:59:48 dist-test-slave-fl7g krb5kdc[17339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:59:48.033934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 17714
I20260926 20:59:48.034017 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:59:48.034221 392 external_mini_cluster.cc:1428] Setting key 115c2aea69248206a9cdf3e6fad9bfb8
I20260926 20:59:48.035422 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.025509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60807 (local address 127.0.98.62:39781)
0926 20:59:48.025591 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:48.025594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:48.025817 (+ 223us) server_negotiation.cc:407] Connection header received
0926 20:59:48.026751 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:48.026753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:48.026764 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:48.026827 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:48.027456 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.027967 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.029315 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.029506 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.029571 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.032113 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:48.032115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:48.032115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:48.032123 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:48.033689 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:48.034218 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:48.034219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:48.034220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:48.034285 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:48.034672 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:48.034677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:48.034677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:48.034865 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:48.034932 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:48.035286 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 20:59:48.035357 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:59:48.035681 17845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.025490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39781 (local address 127.0.98.3:60807)
0926 20:59:48.025691 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:48.025728 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:48.026637 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:48.026895 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:48.026898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:48.027174 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:48.027372 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.027376 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.028086 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.028087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.029211 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.029214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.029591 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.029592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.029721 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.030318 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:59:48.030328 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:48.032008 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:48.033785 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:48.033806 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:48.033808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:48.034114 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:48.034382 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:48.034383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:48.034384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:48.034571 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:48.034936 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:48.034944 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:48.035131 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 20:59:48.035260 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 20:59:48.035402 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":111,"threads_started":1}
I20260926 20:59:48.036314 17843 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39781
I20260926 20:59:48.036478 17843 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:48.036648 17843 heartbeater.cc:511] Master 127.0.98.62:39781 requested a full tablet report, sending...
I20260926 20:59:48.037055 17384 ts_manager.cc:194] Registered new tserver with Master: 1a2f009db6ba4d6195f35435b95d0b62 (127.0.98.3:34657)
I20260926 20:59:48.037416 17384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60807
I20260926 20:59:48.046165 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:59:48.053670 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.047642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.0.98.62:39781)
0926 20:59:48.047705 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:48.047707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:48.047764 (+ 57us) server_negotiation.cc:407] Connection header received
0926 20:59:48.047829 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:48.047831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:48.047842 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:48.047885 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:48.048484 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.048998 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.049763 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.049963 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.050015 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.051362 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:48.051363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:48.051363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:48.051368 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:48.052742 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:48.053049 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:48.053050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:48.053050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:48.053099 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:48.053261 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:48.053261 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:48.053261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:48.053402 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:48.053479 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:48.053548 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 20:59:48.053605 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 20:59:48.055430 17384 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:59:48.057235 17384 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:59:48.064275 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.060156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32992 (local address 127.0.98.3:34657)
0926 20:59:48.060339 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:48.060344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:48.060353 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:59:48.060391 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:48.060394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:48.060437 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:48.060512 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:48.060742 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.061812 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.063205 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.063813 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.063901 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.063970 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:48.064134 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 20:59:48.064178 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":83,"threads_started":1}
I20260926 20:59:48.064493 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.059803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49522 (local address 127.0.98.1:37499)
0926 20:59:48.060046 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:48.060049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:48.060138 (+ 89us) server_negotiation.cc:407] Connection header received
0926 20:59:48.060208 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:48.060211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:48.060230 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:48.060311 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:48.060604 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.061788 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.063129 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.063897 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.063969 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.064025 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:48.064346 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 20:59:48.064393 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":75,"threads_started":1}
I20260926 20:59:48.064493 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.059805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.0.98.2:43035)
0926 20:59:48.059965 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:48.059970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:48.060083 (+ 113us) server_negotiation.cc:407] Connection header received
0926 20:59:48.060471 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:48.060476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:48.060493 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:48.060566 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:48.060805 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.061511 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.063130 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.063898 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.063969 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.064024 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:48.064346 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 20:59:48.064389 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260926 20:59:48.064265 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.059943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34657 (local address 127.0.0.1:32992)
0926 20:59:48.060145 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:48.060155 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:48.060198 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:48.060603 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:48.060604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:48.060605 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:48.060663 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.060666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.061921 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.061922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.063143 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.063144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.063903 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.063904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.063994 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.064008 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 20:59:48.064113 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 20:59:48.064178 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":54,"thread_start_us":62,"threads_started":1}
I20260926 20:59:48.064857 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.059689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37499 (local address 127.0.0.1:49522)
0926 20:59:48.059990 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:48.060025 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:48.060109 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:48.060325 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:48.060327 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:48.060330 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:48.060435 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.060437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.061917 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.061918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.062985 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.062987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.064063 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.064064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.064144 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.064150 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:59:48.064201 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 20:59:48.064254 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":38,"thread_start_us":62,"threads_started":1}
I20260926 20:59:48.065383 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.059687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43035 (local address 127.0.0.1:40344)
0926 20:59:48.059978 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:48.060024 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:48.060325 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:48.060657 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:48.060658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:48.060660 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:48.060724 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.060725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.061621 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.061622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.063045 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.063047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.064065 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.064066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.064144 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.064152 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 20:59:48.064201 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 20:59:48.064266 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":10,"thread_start_us":149,"threads_started":1}
I20260926 20:59:48.065728 17777 tablet_service.cc:1467] Processing CreateTablet for tablet 5f08eccc6c6c4efdbf659453ec2ea0bf (DEFAULT_TABLE table=test-table [id=dd56be950e9b4f60a31c5d3bff1b79f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:48.065732 17509 tablet_service.cc:1467] Processing CreateTablet for tablet 5f08eccc6c6c4efdbf659453ec2ea0bf (DEFAULT_TABLE table=test-table [id=dd56be950e9b4f60a31c5d3bff1b79f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:48.066089 17777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f08eccc6c6c4efdbf659453ec2ea0bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:48.066089 17509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f08eccc6c6c4efdbf659453ec2ea0bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:48.066030 17643 tablet_service.cc:1467] Processing CreateTablet for tablet 5f08eccc6c6c4efdbf659453ec2ea0bf (DEFAULT_TABLE table=test-table [id=dd56be950e9b4f60a31c5d3bff1b79f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:48.066278 17643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f08eccc6c6c4efdbf659453ec2ea0bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:48.068825 17860 tablet_bootstrap.cc:492] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b: Bootstrap starting.
I20260926 20:59:48.069372 17862 tablet_bootstrap.cc:492] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62: Bootstrap starting.
I20260926 20:59:48.069523 17860 tablet_bootstrap.cc:654] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:48.070067 17862 tablet_bootstrap.cc:654] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:48.070158 17860 log.cc:824] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:48.070621 17862 log.cc:824] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:48.070979 17861 tablet_bootstrap.cc:492] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 4cd7b9a40cbc43698a0ec388a5d6c552: Bootstrap starting.
I20260926 20:59:48.071017 17860 tablet_bootstrap.cc:492] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b: No bootstrap required, opened a new log
I20260926 20:59:48.071074 17860 ts_tablet_manager.cc:1397] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:59:48.071377 17862 tablet_bootstrap.cc:492] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62: No bootstrap required, opened a new log
I20260926 20:59:48.071465 17862 ts_tablet_manager.cc:1397] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:59:48.071625 17861 tablet_bootstrap.cc:654] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 4cd7b9a40cbc43698a0ec388a5d6c552: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:48.072170 17861 log.cc:824] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 4cd7b9a40cbc43698a0ec388a5d6c552: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:48.073062 17860 raft_consensus.cc:348] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236d196459c4a98a7cf3a7a418e874b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43035 } } peers { permanent_uuid: "1a2f009db6ba4d6195f35435b95d0b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34657 } } peers { permanent_uuid: "4cd7b9a40cbc43698a0ec388a5d6c552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37499 } }
I20260926 20:59:48.073210 17860 raft_consensus.cc:374] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:48.073103 17862 raft_consensus.cc:348] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236d196459c4a98a7cf3a7a418e874b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43035 } } peers { permanent_uuid: "1a2f009db6ba4d6195f35435b95d0b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34657 } } peers { permanent_uuid: "4cd7b9a40cbc43698a0ec388a5d6c552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37499 } }
I20260926 20:59:48.073231 17860 raft_consensus.cc:729] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a236d196459c4a98a7cf3a7a418e874b, State: Initialized, Role: FOLLOWER
I20260926 20:59:48.073259 17862 raft_consensus.cc:374] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:48.073339 17862 raft_consensus.cc:729] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a2f009db6ba4d6195f35435b95d0b62, State: Initialized, Role: FOLLOWER
I20260926 20:59:48.073391 17860 consensus_queue.cc:260] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236d196459c4a98a7cf3a7a418e874b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43035 } } peers { permanent_uuid: "1a2f009db6ba4d6195f35435b95d0b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34657 } } peers { permanent_uuid: "4cd7b9a40cbc43698a0ec388a5d6c552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37499 } }
I20260926 20:59:48.073464 17862 consensus_queue.cc:260] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236d196459c4a98a7cf3a7a418e874b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43035 } } peers { permanent_uuid: "1a2f009db6ba4d6195f35435b95d0b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34657 } } peers { permanent_uuid: "4cd7b9a40cbc43698a0ec388a5d6c552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37499 } }
I20260926 20:59:48.073696 17860 ts_tablet_manager.cc:1428] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:59:48.073698 17709 heartbeater.cc:503] Master 127.0.98.62:39781 was elected leader, sending a full tablet report...
I20260926 20:59:48.073760 17862 ts_tablet_manager.cc:1428] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:59:48.073770 17843 heartbeater.cc:503] Master 127.0.98.62:39781 was elected leader, sending a full tablet report...
I20260926 20:59:48.074497 17861 tablet_bootstrap.cc:492] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 4cd7b9a40cbc43698a0ec388a5d6c552: No bootstrap required, opened a new log
I20260926 20:59:48.074573 17861 ts_tablet_manager.cc:1397] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 4cd7b9a40cbc43698a0ec388a5d6c552: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 20:59:48.076267 17861 raft_consensus.cc:348] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 4cd7b9a40cbc43698a0ec388a5d6c552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236d196459c4a98a7cf3a7a418e874b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43035 } } peers { permanent_uuid: "1a2f009db6ba4d6195f35435b95d0b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34657 } } peers { permanent_uuid: "4cd7b9a40cbc43698a0ec388a5d6c552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37499 } }
I20260926 20:59:48.076500 17861 raft_consensus.cc:374] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 4cd7b9a40cbc43698a0ec388a5d6c552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:48.076556 17861 raft_consensus.cc:729] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 4cd7b9a40cbc43698a0ec388a5d6c552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd7b9a40cbc43698a0ec388a5d6c552, State: Initialized, Role: FOLLOWER
I20260926 20:59:48.076696 17861 consensus_queue.cc:260] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 4cd7b9a40cbc43698a0ec388a5d6c552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236d196459c4a98a7cf3a7a418e874b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43035 } } peers { permanent_uuid: "1a2f009db6ba4d6195f35435b95d0b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34657 } } peers { permanent_uuid: "4cd7b9a40cbc43698a0ec388a5d6c552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37499 } }
I20260926 20:59:48.076997 17861 ts_tablet_manager.cc:1428] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 4cd7b9a40cbc43698a0ec388a5d6c552: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 20:59:48.077019 17575 heartbeater.cc:503] Master 127.0.98.62:39781 was elected leader, sending a full tablet report...
W20260926 20:59:48.105098 17710 tablet.cc:2369] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:59:48.172016 17867 raft_consensus.cc:482] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:59:48.172188 17867 raft_consensus.cc:504] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236d196459c4a98a7cf3a7a418e874b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43035 } } peers { permanent_uuid: "1a2f009db6ba4d6195f35435b95d0b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34657 } } peers { permanent_uuid: "4cd7b9a40cbc43698a0ec388a5d6c552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37499 } }
I20260926 20:59:48.172623 17867 leader_election.cc:288] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a236d196459c4a98a7cf3a7a418e874b (127.0.98.2:43035), 4cd7b9a40cbc43698a0ec388a5d6c552 (127.0.98.1:37499)
I20260926 20:59:48.175487 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.172919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43907 (local address 127.0.98.1:37499)
0926 20:59:48.172964 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:48.172966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:48.172976 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:59:48.173005 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:48.173005 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:48.173018 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:48.173072 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:48.173264 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.173740 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.174729 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.175114 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.175202 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.175213 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:48.175379 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 20:59:48.175417 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:59:48.175519 17845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.172817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37499 (local address 127.0.98.3:43907)
0926 20:59:48.172878 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:48.172892 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:48.172945 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:48.173093 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:48.173094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:48.173096 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:48.173174 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.173176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.173825 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.173826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.174627 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.174628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.175201 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.175202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.175289 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.175292 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:59:48.175370 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:59:48.175414 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":45}
I20260926 20:59:48.175546 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.172961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57261 (local address 127.0.98.2:43035)
0926 20:59:48.173008 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:48.173011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:48.173088 (+ 77us) server_negotiation.cc:407] Connection header received
0926 20:59:48.173138 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:48.173138 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:48.173151 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:48.173226 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:48.173393 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.173857 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.174821 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.175184 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.175264 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.175274 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:48.175435 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 20:59:48.175472 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:59:48.175635 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.172833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43035 (local address 127.0.98.3:57261)
0926 20:59:48.173007 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:48.173018 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:48.173084 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:48.173229 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:48.173230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:48.173232 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:48.173320 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.173321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.173940 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.173941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.174704 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.174706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.175279 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.175280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.175360 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.175363 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:59:48.175428 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:59:48.175467 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":2,"thread_start_us":107,"threads_started":1}
I20260926 20:59:48.175948 17529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f08eccc6c6c4efdbf659453ec2ea0bf" candidate_uuid: "1a2f009db6ba4d6195f35435b95d0b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd7b9a40cbc43698a0ec388a5d6c552" is_pre_election: true
I20260926 20:59:48.176132 17529 raft_consensus.cc:2454] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 4cd7b9a40cbc43698a0ec388a5d6c552 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a2f009db6ba4d6195f35435b95d0b62 in term 0.
I20260926 20:59:48.176083 17663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f08eccc6c6c4efdbf659453ec2ea0bf" candidate_uuid: "1a2f009db6ba4d6195f35435b95d0b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a236d196459c4a98a7cf3a7a418e874b" is_pre_election: true
I20260926 20:59:48.176235 17663 raft_consensus.cc:2454] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a2f009db6ba4d6195f35435b95d0b62 in term 0.
I20260926 20:59:48.176442 17733 leader_election.cc:302] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a2f009db6ba4d6195f35435b95d0b62, 4cd7b9a40cbc43698a0ec388a5d6c552; no voters:
I20260926 20:59:48.176587 17867 raft_consensus.cc:2781] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:59:48.176667 17867 raft_consensus.cc:482] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:59:48.176702 17867 raft_consensus.cc:3037] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:48.177448 17867 raft_consensus.cc:504] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236d196459c4a98a7cf3a7a418e874b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43035 } } peers { permanent_uuid: "1a2f009db6ba4d6195f35435b95d0b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34657 } } peers { permanent_uuid: "4cd7b9a40cbc43698a0ec388a5d6c552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37499 } }
I20260926 20:59:48.177599 17867 leader_election.cc:288] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [CANDIDATE]: Term 1 election: Requested vote from peers a236d196459c4a98a7cf3a7a418e874b (127.0.98.2:43035), 4cd7b9a40cbc43698a0ec388a5d6c552 (127.0.98.1:37499)
I20260926 20:59:48.177802 17663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f08eccc6c6c4efdbf659453ec2ea0bf" candidate_uuid: "1a2f009db6ba4d6195f35435b95d0b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a236d196459c4a98a7cf3a7a418e874b"
I20260926 20:59:48.177881 17663 raft_consensus.cc:3037] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:48.178587 17663 raft_consensus.cc:2454] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a2f009db6ba4d6195f35435b95d0b62 in term 1.
I20260926 20:59:48.178714 17529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f08eccc6c6c4efdbf659453ec2ea0bf" candidate_uuid: "1a2f009db6ba4d6195f35435b95d0b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cd7b9a40cbc43698a0ec388a5d6c552"
I20260926 20:59:48.178787 17731 leader_election.cc:302] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a2f009db6ba4d6195f35435b95d0b62, a236d196459c4a98a7cf3a7a418e874b; no voters:
I20260926 20:59:48.178804 17529 raft_consensus.cc:3037] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 4cd7b9a40cbc43698a0ec388a5d6c552 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:48.178905 17867 raft_consensus.cc:2781] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:59:48.179065 17867 raft_consensus.cc:686] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [term 1 LEADER]: Becoming Leader. State: Replica: 1a2f009db6ba4d6195f35435b95d0b62, State: Running, Role: LEADER
I20260926 20:59:48.179198 17867 consensus_queue.cc:237] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236d196459c4a98a7cf3a7a418e874b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43035 } } peers { permanent_uuid: "1a2f009db6ba4d6195f35435b95d0b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34657 } } peers { permanent_uuid: "4cd7b9a40cbc43698a0ec388a5d6c552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37499 } }
I20260926 20:59:48.179680 17529 raft_consensus.cc:2454] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 4cd7b9a40cbc43698a0ec388a5d6c552 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a2f009db6ba4d6195f35435b95d0b62 in term 1.
I20260926 20:59:48.180099 17384 catalog_manager.cc:5495] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a2f009db6ba4d6195f35435b95d0b62 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1a2f009db6ba4d6195f35435b95d0b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a236d196459c4a98a7cf3a7a418e874b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a2f009db6ba4d6195f35435b95d0b62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cd7b9a40cbc43698a0ec388a5d6c552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37499 } health_report { overall_health: UNKNOWN } } }
W20260926 20:59:48.189937 17576 tablet.cc:2369] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 4cd7b9a40cbc43698a0ec388a5d6c552: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:59:48.213865 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.211517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.0.98.3:34657)
0926 20:59:48.211584 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:48.211586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:48.211602 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:59:48.211638 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:48.211639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:48.211656 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:48.211718 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:59:48.211861 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.212330 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.213135 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.213323 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.213377 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.213506 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:59:48.213602 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:59:48.213662 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:48.213693 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 20:59:48.213708 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:59:48.215636 17663 raft_consensus.cc:1261] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b [term 1 FOLLOWER]: Refusing update from remote peer 1a2f009db6ba4d6195f35435b95d0b62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:59:48.215644 17529 raft_consensus.cc:1261] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 4cd7b9a40cbc43698a0ec388a5d6c552 [term 1 FOLLOWER]: Refusing update from remote peer 1a2f009db6ba4d6195f35435b95d0b62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:59:48.216030 17867 consensus_queue.cc:1035] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd7b9a40cbc43698a0ec388a5d6c552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:59:48.216157 17870 consensus_queue.cc:1035] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a236d196459c4a98a7cf3a7a418e874b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:59:48.218524 17873 mvcc.cc:205] Tried to move back new op lower bound from 7333709366128705536 to 7333709365982838784. Current Snapshot: MvccSnapshot[applied={T|T < 7333709366128705536}]
I20260926 20:59:48.218693 17875 mvcc.cc:205] Tried to move back new op lower bound from 7333709366128705536 to 7333709365982838784. Current Snapshot: MvccSnapshot[applied={T|T < 7333709366128705536}]
I20260926 20:59:48.218727 17874 mvcc.cc:205] Tried to move back new op lower bound from 7333709366128705536 to 7333709365982838784. Current Snapshot: MvccSnapshot[applied={T|T < 7333709366128705536}]
I20260926 20:59:48.221007 17384 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:59:48.221105 17384 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:59:48.221894 17384 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27a783c3259546c4b98376e17bf1adfd: Sending DeleteTablet for 3 replicas of tablet 5f08eccc6c6c4efdbf659453ec2ea0bf
I20260926 20:59:48.222252 17509 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f08eccc6c6c4efdbf659453ec2ea0bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49522
I20260926 20:59:48.222360 17643 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f08eccc6c6c4efdbf659453ec2ea0bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40344
I20260926 20:59:48.222518 17883 tablet_replica.cc:330] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 4cd7b9a40cbc43698a0ec388a5d6c552: stopping tablet replica
I20260926 20:59:48.222494 17777 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f08eccc6c6c4efdbf659453ec2ea0bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32992
I20260926 20:59:48.222623 17883 raft_consensus.cc:2229] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 4cd7b9a40cbc43698a0ec388a5d6c552 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:59:48.222743 17883 raft_consensus.cc:2258] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 4cd7b9a40cbc43698a0ec388a5d6c552 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:59:48.222755 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 17446
I20260926 20:59:48.225348 17885 tablet_replica.cc:330] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62: stopping tablet replica
I20260926 20:59:48.225512 17884 tablet_replica.cc:330] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b: stopping tablet replica
I20260926 20:59:48.225672 17884 raft_consensus.cc:2229] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:59:48.225796 17885 raft_consensus.cc:2229] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:59:48.225971 17885 raft_consensus.cc:2258] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:59:48.226128 17884 raft_consensus.cc:2258] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:59:48.226788 17884 ts_tablet_manager.cc:1905] T 5f08eccc6c6c4efdbf659453ec2ea0bf P a236d196459c4a98a7cf3a7a418e874b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:59:48.226814 17885 ts_tablet_manager.cc:1905] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:59:48.228698 17885 ts_tablet_manager.cc:1918] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:59:48.228849 17885 log.cc:1196] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/5f08eccc6c6c4efdbf659453ec2ea0bf
W20260926 20:59:48.228909 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37499 (error 108)
I20260926 20:59:48.229125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 17580
I20260926 20:59:48.230020 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.229871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37499 (local address 127.0.0.1:49532)
0926 20:59:48.229921 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:48.229974 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 20:59:48.230464 17371 catalog_manager.cc:4584] TS 4cd7b9a40cbc43698a0ec388a5d6c552 (127.0.98.1:37499): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f08eccc6c6c4efdbf659453ec2ea0bf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37499: connect: Connection refused (error 111)
I20260926 20:59:48.232967 17885 ts_tablet_manager.cc:1939] T 5f08eccc6c6c4efdbf659453ec2ea0bf P 1a2f009db6ba4d6195f35435b95d0b62: Deleting consensus metadata
I20260926 20:59:48.233433 17369 catalog_manager.cc:4841] TS 1a2f009db6ba4d6195f35435b95d0b62 (127.0.98.3:34657): tablet 5f08eccc6c6c4efdbf659453ec2ea0bf (table test-table [id=dd56be950e9b4f60a31c5d3bff1b79f9]) successfully deleted
I20260926 20:59:48.235029 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.234908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43035 (local address 127.0.0.1:40352)
0926 20:59:48.234967 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:48.235001 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 20:59:48.235692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 17714
I20260926 20:59:48.239943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 17352
2026-09-26T20:59:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2058 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T20:59:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:59:48Z Disabled control of system clock
I20260926 20:59:48.268748 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39425
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:59:48.350311 17890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:48.350541 17890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:48.350561 17890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:48.351655 17890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:59:48.351687 17890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:48.351701 17890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:48.351712 17890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:59:48.351724 17890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:59:48.353119 17890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:48.353328 17890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:48.353566 17890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:48.358690 17896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:48.358703 17895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:48.358691 17898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:48.358711 17890 server_base.cc:1034] running on GCE node
I20260926 20:59:48.359251 17890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:48.359557 17890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:48.360700 17890 hybrid_clock.cc:648] HybridClock initialized: now 1790456388360666 us; error 55 us; skew 500 ppm
I20260926 20:59:48.362102 17890 webserver.cc:460] Webserver started at http://127.0.98.62:43111/ using document root <none> and password file <none>
I20260926 20:59:48.362378 17890 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:48.362429 17890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:48.362609 17890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:48.363713 17890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "5a0fbfe15d5f44b4bb8ed99e18b32b62"
format_stamp: "Formatted at 2026-09-26 20:59:48 on dist-test-slave-fl7g"
server_key: "a49d3e76ed3dfc79c09d3f358ddc156c"
server_key_iv: "45070eec4ae8e7902466f7b78a971bf1"
server_key_version: "encryptionkey@0"
I20260926 20:59:48.364097 17890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "5a0fbfe15d5f44b4bb8ed99e18b32b62"
format_stamp: "Formatted at 2026-09-26 20:59:48 on dist-test-slave-fl7g"
server_key: "a49d3e76ed3dfc79c09d3f358ddc156c"
server_key_iv: "45070eec4ae8e7902466f7b78a971bf1"
server_key_version: "encryptionkey@0"
I20260926 20:59:48.365468 17890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:59:48.366299 17904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:48.366568 17890 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:59:48.366683 17890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "5a0fbfe15d5f44b4bb8ed99e18b32b62"
format_stamp: "Formatted at 2026-09-26 20:59:48 on dist-test-slave-fl7g"
server_key: "a49d3e76ed3dfc79c09d3f358ddc156c"
server_key_iv: "45070eec4ae8e7902466f7b78a971bf1"
server_key_version: "encryptionkey@0"
I20260926 20:59:48.366777 17890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:48.381081 17890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:48.381461 17890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:48.381630 17890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:48.386010 17890 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39425
I20260926 20:59:48.386063 17956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39425 every 8 connection(s)
I20260926 20:59:48.386458 17890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:59:48.387423 17957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:48.389711 17957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62: Bootstrap starting.
I20260926 20:59:48.390240 17957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:48.390482 17957 log.cc:824] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:48.391161 17957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62: No bootstrap required, opened a new log
I20260926 20:59:48.392903 17957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fbfe15d5f44b4bb8ed99e18b32b62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } }
I20260926 20:59:48.393043 17957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:48.393065 17957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0fbfe15d5f44b4bb8ed99e18b32b62, State: Initialized, Role: FOLLOWER
I20260926 20:59:48.393224 17957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fbfe15d5f44b4bb8ed99e18b32b62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } }
I20260926 20:59:48.393298 17957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:59:48.393275 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 17890
I20260926 20:59:48.393344 17957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:59:48.393399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:59:48.393395 17957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:48.393590 392 external_mini_cluster.cc:1428] Setting key 8eb7145cc717d653eab7151fa7f63f46
I20260926 20:59:48.394155 17957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fbfe15d5f44b4bb8ed99e18b32b62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } }
I20260926 20:59:48.394272 17957 leader_election.cc:302] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a0fbfe15d5f44b4bb8ed99e18b32b62; no voters:
I20260926 20:59:48.394433 17957 leader_election.cc:288] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:59:48.394464 17962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:59:48.394579 17962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62 [term 1 LEADER]: Becoming Leader. State: Replica: 5a0fbfe15d5f44b4bb8ed99e18b32b62, State: Running, Role: LEADER
I20260926 20:59:48.394692 17957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:59:48.394716 17962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fbfe15d5f44b4bb8ed99e18b32b62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } }
I20260926 20:59:48.395900 17963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a0fbfe15d5f44b4bb8ed99e18b32b62. Latest consensus state: current_term: 1 leader_uuid: "5a0fbfe15d5f44b4bb8ed99e18b32b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fbfe15d5f44b4bb8ed99e18b32b62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } } }
I20260926 20:59:48.395998 17963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62 [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:48.396202 17977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:59:48.396360 17964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a0fbfe15d5f44b4bb8ed99e18b32b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fbfe15d5f44b4bb8ed99e18b32b62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } } }
I20260926 20:59:48.396489 17964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62 [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:48.396915 17977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:59:48.398543 17977 catalog_manager.cc:1347] Generated new cluster ID: 02f3bf1418a84aa6805e17bfbe995939
I20260926 20:59:48.398655 17977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:59:48.400239 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.394124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45158 (local address 127.0.98.62:39425)
0926 20:59:48.394410 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:48.394419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:48.394449 (+ 30us) server_negotiation.cc:407] Connection header received
0926 20:59:48.394600 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:48.394608 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:48.394677 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:48.395075 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:48.395169 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.396947 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.397896 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.398501 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.398635 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.399415 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:48.399419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:48.399420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:48.399626 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:48.399668 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:48.399683 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:48.399758 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:48.399824 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 20:59:48.400008 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":105,"threads_started":1}
I20260926 20:59:48.408667 17977 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:59:48.409288 17977 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:59:48.418483 17977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62: Generated new TSK 0
I20260926 20:59:48.418709 17977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:59:48.422807 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39425
--builtin_ntp_servers=127.0.98.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:59:48.505975 17981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:48.506223 17981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:48.506263 17981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:48.507530 17981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:48.507586 17981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:48.507637 17981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:59:48.509120 17981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:48.509389 17981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:48.509713 17981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:48.515823 17986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:48.515827 17987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:48.515944 17989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:48.516108 17981 server_base.cc:1034] running on GCE node
I20260926 20:59:48.516321 17981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:48.516623 17981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:48.517812 17981 hybrid_clock.cc:648] HybridClock initialized: now 1790456388517729 us; error 98 us; skew 500 ppm
I20260926 20:59:48.519375 17981 webserver.cc:460] Webserver started at http://127.0.98.1:46369/ using document root <none> and password file <none>
I20260926 20:59:48.519639 17981 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:48.519698 17981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:48.519822 17981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:48.521220 17981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "727d1543b7644ab99894c46ea42ce4b5"
format_stamp: "Formatted at 2026-09-26 20:59:48 on dist-test-slave-fl7g"
server_key: "fcd69c28cac0d86ba06f15f9126e917a"
server_key_iv: "ce543ef206f939d1a9e251887c4525c7"
server_key_version: "encryptionkey@0"
I20260926 20:59:48.521602 17981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "727d1543b7644ab99894c46ea42ce4b5"
format_stamp: "Formatted at 2026-09-26 20:59:48 on dist-test-slave-fl7g"
server_key: "fcd69c28cac0d86ba06f15f9126e917a"
server_key_iv: "ce543ef206f939d1a9e251887c4525c7"
server_key_version: "encryptionkey@0"
I20260926 20:59:48.523195 17981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:59:48.524168 17995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:48.524359 17981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:48.524492 17981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "727d1543b7644ab99894c46ea42ce4b5"
format_stamp: "Formatted at 2026-09-26 20:59:48 on dist-test-slave-fl7g"
server_key: "fcd69c28cac0d86ba06f15f9126e917a"
server_key_iv: "ce543ef206f939d1a9e251887c4525c7"
server_key_version: "encryptionkey@0"
I20260926 20:59:48.524576 17981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:48.532356 17981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:48.532691 17981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:48.532804 17981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:48.532979 17981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:48.533303 17981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:48.533342 17981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:48.533365 17981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:48.533409 17981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:48.539062 17981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46009
I20260926 20:59:48.539163 18108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46009 every 8 connection(s)
I20260926 20:59:48.539505 17981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:59:48.545359 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.540361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45849 (local address 127.0.98.62:39425)
0926 20:59:48.540506 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:48.540509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:48.540710 (+ 201us) server_negotiation.cc:407] Connection header received
0926 20:59:48.541258 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:48.541259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:48.541274 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:48.541340 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:48.542064 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.542596 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.543706 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.543905 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.543973 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.545083 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:48.545084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:48.545084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:48.545089 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:48.545106 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:48.545110 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:48.545170 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:48.545260 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 20:59:48.545293 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 20:59:48.545621 18111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.540346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39425 (local address 127.0.98.1:45849)
0926 20:59:48.540566 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:48.540611 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:48.541137 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:48.541425 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:48.541428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:48.541437 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:48.541951 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.541955 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.542702 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.542703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.543514 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.543516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.544020 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.544021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.544155 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.544845 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:59:48.544845 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:48.544877 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:59:48.544926 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:48.545180 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:48.545185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:48.545187 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:59:48.545243 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:59:48.545341 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":109,"threads_started":1}
I20260926 20:59:48.546191 18109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39425
I20260926 20:59:48.546312 18109 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:48.546502 18109 heartbeater.cc:511] Master 127.0.98.62:39425 requested a full tablet report, sending...
I20260926 20:59:48.547055 17921 ts_manager.cc:194] Registered new tserver with Master: 727d1543b7644ab99894c46ea42ce4b5 (127.0.98.1:46009)
I20260926 20:59:48.547694 17921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:45849
I20260926 20:59:48.548024 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 17981
I20260926 20:59:48.548122 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:59:48.548367 392 external_mini_cluster.cc:1428] Setting key d6fcb602e0eaf2418a453fd33844bb50
I20260926 20:59:48.549309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39425
--builtin_ntp_servers=127.0.98.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:59:48.633220 18112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:48.633431 18112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:48.633455 18112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:48.634646 18112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:48.634687 18112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:48.634713 18112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:59:48.636054 18112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:48.636313 18112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:48.636679 18112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:48.641798 18117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:48.641855 18118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:48.641899 18120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:48.641971 18112 server_base.cc:1034] running on GCE node
I20260926 20:59:48.642334 18112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:48.642612 18112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:48.643770 18112 hybrid_clock.cc:648] HybridClock initialized: now 1790456388643742 us; error 43 us; skew 500 ppm
I20260926 20:59:48.645147 18112 webserver.cc:460] Webserver started at http://127.0.98.2:40747/ using document root <none> and password file <none>
I20260926 20:59:48.645413 18112 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:48.645460 18112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:48.645634 18112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:48.646767 18112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "cccc12122ec14318a6c51e2ed44cb3dc"
format_stamp: "Formatted at 2026-09-26 20:59:48 on dist-test-slave-fl7g"
server_key: "e37fd160c93a0feb00af62624b3f17ee"
server_key_iv: "f844103350d537ccf47b6a2856f68e3e"
server_key_version: "encryptionkey@0"
I20260926 20:59:48.647136 18112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cccc12122ec14318a6c51e2ed44cb3dc"
format_stamp: "Formatted at 2026-09-26 20:59:48 on dist-test-slave-fl7g"
server_key: "e37fd160c93a0feb00af62624b3f17ee"
server_key_iv: "f844103350d537ccf47b6a2856f68e3e"
server_key_version: "encryptionkey@0"
I20260926 20:59:48.648644 18112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:59:48.649448 18126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:48.649675 18112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:48.649751 18112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "cccc12122ec14318a6c51e2ed44cb3dc"
format_stamp: "Formatted at 2026-09-26 20:59:48 on dist-test-slave-fl7g"
server_key: "e37fd160c93a0feb00af62624b3f17ee"
server_key_iv: "f844103350d537ccf47b6a2856f68e3e"
server_key_version: "encryptionkey@0"
I20260926 20:59:48.649843 18112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:48.661103 18112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:48.661443 18112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:48.661607 18112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:48.661857 18112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:48.662226 18112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:48.662285 18112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:48.662328 18112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:48.662355 18112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:48.668272 18112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45357
I20260926 20:59:48.668315 18239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45357 every 8 connection(s)
I20260926 20:59:48.668692 18112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:59:48.674000 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.669504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52603 (local address 127.0.98.62:39425)
0926 20:59:48.669568 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:48.669570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:48.669846 (+ 276us) server_negotiation.cc:407] Connection header received
0926 20:59:48.670289 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:48.670290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:48.670302 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:48.670371 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:48.671031 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.671529 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.672653 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.672849 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.672917 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.673732 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:48.673733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:48.673733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:48.673738 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:48.673755 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:48.673760 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:48.673816 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:48.673911 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 20:59:48.673938 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:59:48.674227 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.669496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39425 (local address 127.0.98.2:52603)
0926 20:59:48.669678 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:48.669715 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:48.670198 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:48.670439 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:48.670442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:48.670450 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:48.670957 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.670961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.671654 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.671655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.672552 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.672555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.672935 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.672936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.673052 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.673584 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:59:48.673584 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:48.673614 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:59:48.673636 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:48.673819 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:48.673822 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:48.673824 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:59:48.673874 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 20:59:48.673962 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260926 20:59:48.674468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 18112
I20260926 20:59:48.674568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:59:48.674763 18240 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39425
I20260926 20:59:48.674834 392 external_mini_cluster.cc:1428] Setting key c955fb4ae31025c12a85484861153dc4
I20260926 20:59:48.674865 18240 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:48.675066 18240 heartbeater.cc:511] Master 127.0.98.62:39425 requested a full tablet report, sending...
I20260926 20:59:48.675442 17920 ts_manager.cc:194] Registered new tserver with Master: cccc12122ec14318a6c51e2ed44cb3dc (127.0.98.2:45357)
I20260926 20:59:48.675812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39425
--builtin_ntp_servers=127.0.98.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:59:48.675887 17920 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:52603
W20260926 20:59:48.759002 18243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:48.759227 18243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:48.759249 18243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:48.760372 18243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:48.760471 18243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:48.760565 18243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:59:48.761998 18243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:48.762265 18243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:48.762666 18243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:48.768735 18249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:48.768774 18248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:48.768832 18251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:48.768790 18243 server_base.cc:1034] running on GCE node
I20260926 20:59:48.769170 18243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:48.769428 18243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:48.770602 18243 hybrid_clock.cc:648] HybridClock initialized: now 1790456388770594 us; error 38 us; skew 500 ppm
I20260926 20:59:48.771926 18243 webserver.cc:460] Webserver started at http://127.0.98.3:43327/ using document root <none> and password file <none>
I20260926 20:59:48.772151 18243 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:48.772195 18243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:48.772287 18243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:48.773425 18243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "0659b47504b846339f5a55082a7e113e"
format_stamp: "Formatted at 2026-09-26 20:59:48 on dist-test-slave-fl7g"
server_key: "29faff2fca23b2e8f2a539921b0c563e"
server_key_iv: "c1b15b93205423f4d1b251cce4685cc4"
server_key_version: "encryptionkey@0"
I20260926 20:59:48.773736 18243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0659b47504b846339f5a55082a7e113e"
format_stamp: "Formatted at 2026-09-26 20:59:48 on dist-test-slave-fl7g"
server_key: "29faff2fca23b2e8f2a539921b0c563e"
server_key_iv: "c1b15b93205423f4d1b251cce4685cc4"
server_key_version: "encryptionkey@0"
I20260926 20:59:48.775012 18243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:59:48.775686 18257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:48.775867 18243 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:59:48.775964 18243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "0659b47504b846339f5a55082a7e113e"
format_stamp: "Formatted at 2026-09-26 20:59:48 on dist-test-slave-fl7g"
server_key: "29faff2fca23b2e8f2a539921b0c563e"
server_key_iv: "c1b15b93205423f4d1b251cce4685cc4"
server_key_version: "encryptionkey@0"
I20260926 20:59:48.776054 18243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:48.785976 18243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:48.786330 18243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:48.786494 18243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:48.786737 18243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:48.787082 18243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:48.787145 18243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:48.787206 18243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:48.787236 18243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:48.793267 18243 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38719
I20260926 20:59:48.793323 18370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38719 every 8 connection(s)
I20260926 20:59:48.793692 18243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:59:48.799285 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.794512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58453 (local address 127.0.98.62:39425)
0926 20:59:48.794580 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:48.794582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:48.794913 (+ 331us) server_negotiation.cc:407] Connection header received
0926 20:59:48.795485 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:48.795486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:48.795498 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:48.795549 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:48.796378 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.796946 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.797989 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.798166 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.798216 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.799049 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:48.799050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:48.799051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:48.799054 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:48.799067 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:48.799071 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:48.799115 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:48.799201 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 20:59:48.799224 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:59:48.799526 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.794517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39425 (local address 127.0.98.3:58453)
0926 20:59:48.794770 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:48.794813 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:48.795394 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:48.795611 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:48.795614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:48.795623 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:48.796261 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.796266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.797100 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.797101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.797903 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.797905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.798242 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.798243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.798352 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.798910 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:59:48.798911 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:48.798939 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:59:48.798962 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:48.799130 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:48.799133 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:48.799136 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:59:48.799190 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:59:48.799274 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260926 20:59:48.800115 18371 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39425
I20260926 20:59:48.800218 18371 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:48.800453 18371 heartbeater.cc:511] Master 127.0.98.62:39425 requested a full tablet report, sending...
I20260926 20:59:48.800673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 18243
I20260926 20:59:48.800807 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:59:48.800938 17920 ts_manager.cc:194] Registered new tserver with Master: 0659b47504b846339f5a55082a7e113e (127.0.98.3:38719)
I20260926 20:59:48.801054 392 external_mini_cluster.cc:1428] Setting key 03d0d505e00998c2d88f13b831267c14
I20260926 20:59:48.801437 17920 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:58453
I20260926 20:59:48.801954 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:59:48.806313 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.803346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45168 (local address 127.0.98.62:39425)
0926 20:59:48.803431 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:48.803434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:48.803553 (+ 119us) server_negotiation.cc:407] Connection header received
0926 20:59:48.803628 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:48.803629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:48.803641 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:48.803696 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:48.803872 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.804337 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.805143 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.805350 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.805427 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.806053 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:48.806054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:48.806055 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:48.806059 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:48.806074 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:48.806078 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:48.806126 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:48.806203 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 20:59:48.806225 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:59:48.807889 17920 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:59:48.809471 17920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:59:48.815557 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.811590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.0.98.3:38719)
0926 20:59:48.812053 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:48.812057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:48.812065 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:59:48.812101 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:48.812103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:48.812118 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:48.812213 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:48.812368 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.813153 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.814554 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.815143 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.815234 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.815336 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:48.815423 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 20:59:48.815461 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":90,"threads_started":1}
I20260926 20:59:48.815580 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.811496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38719 (local address 127.0.0.1:54126)
0926 20:59:48.811663 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:48.811704 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:48.811769 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:48.812218 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:48.812219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:48.812221 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:48.812286 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.812288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.813244 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.813245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.814456 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.814459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.815235 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.815236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.815311 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.815324 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 20:59:48.815399 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:59:48.815456 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260926 20:59:48.815591 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.811496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45357 (local address 127.0.0.1:47920)
0926 20:59:48.811859 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:48.811869 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:48.811929 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:48.812128 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:48.812132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:48.812134 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:48.812204 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.812206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.813599 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.813601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.814472 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.814474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.815341 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.815342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.815403 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.815408 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:59:48.815438 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 20:59:48.815480 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":41,"thread_start_us":60,"threads_started":1}
I20260926 20:59:48.815896 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.811698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46009 (local address 127.0.0.1:36906)
0926 20:59:48.811922 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:48.811932 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:48.811977 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:48.812357 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:48.812357 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:48.812359 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:48.812391 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.812392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.813645 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.813646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.814698 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:48.814700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.815671 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.815672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:48.815735 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.815739 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:59:48.815796 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 20:59:48.815832 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":2,"thread_start_us":77,"threads_started":1}
I20260926 20:59:48.815939 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.811900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36906 (local address 127.0.98.1:46009)
0926 20:59:48.812125 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:48.812129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:48.812139 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:59:48.812175 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:48.812177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:48.812192 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:48.812290 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:48.812493 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.813556 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.814790 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.815493 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.815556 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.815636 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:48.815827 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 20:59:48.815868 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":87,"threads_started":1}
I20260926 20:59:48.816069 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:48.811688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47920 (local address 127.0.98.2:45357)
0926 20:59:48.811846 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:48.811850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:48.811917 (+ 67us) server_negotiation.cc:407] Connection header received
0926 20:59:48.811997 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:48.812000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:48.812017 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:48.812101 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:48.812303 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.813491 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.814565 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:48.815238 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:48.815807 (+ 569us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:48.815895 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:48.815922 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 20:59:48.815964 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260926 20:59:48.816817 18043 tablet_service.cc:1467] Processing CreateTablet for tablet d68e5efdef83464da1579819f928b0d6 (DEFAULT_TABLE table=test-table [id=8a32f78448f945a8b6ad4856c96942e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:48.816792 18305 tablet_service.cc:1467] Processing CreateTablet for tablet d68e5efdef83464da1579819f928b0d6 (DEFAULT_TABLE table=test-table [id=8a32f78448f945a8b6ad4856c96942e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:48.816846 18174 tablet_service.cc:1467] Processing CreateTablet for tablet d68e5efdef83464da1579819f928b0d6 (DEFAULT_TABLE table=test-table [id=8a32f78448f945a8b6ad4856c96942e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:48.817181 18043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68e5efdef83464da1579819f928b0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:48.817183 18305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68e5efdef83464da1579819f928b0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:48.817183 18174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68e5efdef83464da1579819f928b0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:48.819886 18385 tablet_bootstrap.cc:492] T d68e5efdef83464da1579819f928b0d6 P 727d1543b7644ab99894c46ea42ce4b5: Bootstrap starting.
I20260926 20:59:48.820484 18387 tablet_bootstrap.cc:492] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc: Bootstrap starting.
I20260926 20:59:48.820722 18385 tablet_bootstrap.cc:654] T d68e5efdef83464da1579819f928b0d6 P 727d1543b7644ab99894c46ea42ce4b5: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:48.821067 18385 log.cc:824] T d68e5efdef83464da1579819f928b0d6 P 727d1543b7644ab99894c46ea42ce4b5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:48.821204 18387 tablet_bootstrap.cc:654] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:48.821529 18387 log.cc:824] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:48.821864 18385 tablet_bootstrap.cc:492] T d68e5efdef83464da1579819f928b0d6 P 727d1543b7644ab99894c46ea42ce4b5: No bootstrap required, opened a new log
I20260926 20:59:48.821878 18386 tablet_bootstrap.cc:492] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e: Bootstrap starting.
I20260926 20:59:48.821938 18385 ts_tablet_manager.cc:1397] T d68e5efdef83464da1579819f928b0d6 P 727d1543b7644ab99894c46ea42ce4b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:59:48.822207 18387 tablet_bootstrap.cc:492] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc: No bootstrap required, opened a new log
I20260926 20:59:48.822288 18387 ts_tablet_manager.cc:1397] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:59:48.822507 18386 tablet_bootstrap.cc:654] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:48.822860 18386 log.cc:824] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:48.823591 18386 tablet_bootstrap.cc:492] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e: No bootstrap required, opened a new log
I20260926 20:59:48.823654 18386 ts_tablet_manager.cc:1397] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:59:48.824276 18387 raft_consensus.cc:348] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cccc12122ec14318a6c51e2ed44cb3dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45357 } } peers { permanent_uuid: "0659b47504b846339f5a55082a7e113e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38719 } } peers { permanent_uuid: "727d1543b7644ab99894c46ea42ce4b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46009 } }
I20260926 20:59:48.824460 18387 raft_consensus.cc:374] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:48.824497 18387 raft_consensus.cc:729] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cccc12122ec14318a6c51e2ed44cb3dc, State: Initialized, Role: FOLLOWER
I20260926 20:59:48.824625 18387 consensus_queue.cc:260] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cccc12122ec14318a6c51e2ed44cb3dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45357 } } peers { permanent_uuid: "0659b47504b846339f5a55082a7e113e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38719 } } peers { permanent_uuid: "727d1543b7644ab99894c46ea42ce4b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46009 } }
I20260926 20:59:48.824890 18387 ts_tablet_manager.cc:1428] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 20:59:48.824916 18240 heartbeater.cc:503] Master 127.0.98.62:39425 was elected leader, sending a full tablet report...
I20260926 20:59:48.825028 18385 raft_consensus.cc:348] T d68e5efdef83464da1579819f928b0d6 P 727d1543b7644ab99894c46ea42ce4b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cccc12122ec14318a6c51e2ed44cb3dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45357 } } peers { permanent_uuid: "0659b47504b846339f5a55082a7e113e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38719 } } peers { permanent_uuid: "727d1543b7644ab99894c46ea42ce4b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46009 } }
I20260926 20:59:48.825177 18385 raft_consensus.cc:374] T d68e5efdef83464da1579819f928b0d6 P 727d1543b7644ab99894c46ea42ce4b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:48.825210 18385 raft_consensus.cc:729] T d68e5efdef83464da1579819f928b0d6 P 727d1543b7644ab99894c46ea42ce4b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 727d1543b7644ab99894c46ea42ce4b5, State: Initialized, Role: FOLLOWER
I20260926 20:59:48.825337 18385 consensus_queue.cc:260] T d68e5efdef83464da1579819f928b0d6 P 727d1543b7644ab99894c46ea42ce4b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cccc12122ec14318a6c51e2ed44cb3dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45357 } } peers { permanent_uuid: "0659b47504b846339f5a55082a7e113e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38719 } } peers { permanent_uuid: "727d1543b7644ab99894c46ea42ce4b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46009 } }
I20260926 20:59:48.825649 18109 heartbeater.cc:503] Master 127.0.98.62:39425 was elected leader, sending a full tablet report...
I20260926 20:59:48.825596 18385 ts_tablet_manager.cc:1428] T d68e5efdef83464da1579819f928b0d6 P 727d1543b7644ab99894c46ea42ce4b5: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 20:59:48.826321 18386 raft_consensus.cc:348] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cccc12122ec14318a6c51e2ed44cb3dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45357 } } peers { permanent_uuid: "0659b47504b846339f5a55082a7e113e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38719 } } peers { permanent_uuid: "727d1543b7644ab99894c46ea42ce4b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46009 } }
I20260926 20:59:48.826481 18386 raft_consensus.cc:374] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:48.826516 18386 raft_consensus.cc:729] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0659b47504b846339f5a55082a7e113e, State: Initialized, Role: FOLLOWER
I20260926 20:59:48.826627 18386 consensus_queue.cc:260] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cccc12122ec14318a6c51e2ed44cb3dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45357 } } peers { permanent_uuid: "0659b47504b846339f5a55082a7e113e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38719 } } peers { permanent_uuid: "727d1543b7644ab99894c46ea42ce4b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46009 } }
I20260926 20:59:48.826861 18386 ts_tablet_manager.cc:1428] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:59:48.826960 18371 heartbeater.cc:503] Master 127.0.98.62:39425 was elected leader, sending a full tablet report...
W20260926 20:59:48.919234 18241 tablet.cc:2369] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:59:49.040263 18110 tablet.cc:2369] T d68e5efdef83464da1579819f928b0d6 P 727d1543b7644ab99894c46ea42ce4b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:59:49.044251 18372 tablet.cc:2369] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:59:49.293987 18391 raft_consensus.cc:482] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:59:49.294034 18393 raft_consensus.cc:482] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:59:49.294153 18391 raft_consensus.cc:504] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cccc12122ec14318a6c51e2ed44cb3dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45357 } } peers { permanent_uuid: "0659b47504b846339f5a55082a7e113e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38719 } } peers { permanent_uuid: "727d1543b7644ab99894c46ea42ce4b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46009 } }
I20260926 20:59:49.294162 18393 raft_consensus.cc:504] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cccc12122ec14318a6c51e2ed44cb3dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45357 } } peers { permanent_uuid: "0659b47504b846339f5a55082a7e113e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38719 } } peers { permanent_uuid: "727d1543b7644ab99894c46ea42ce4b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46009 } }
I20260926 20:59:49.294621 18391 leader_election.cc:288] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0659b47504b846339f5a55082a7e113e (127.0.98.3:38719), 727d1543b7644ab99894c46ea42ce4b5 (127.0.98.1:46009)
I20260926 20:59:49.294589 18393 leader_election.cc:288] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cccc12122ec14318a6c51e2ed44cb3dc (127.0.98.2:45357), 727d1543b7644ab99894c46ea42ce4b5 (127.0.98.1:46009)
I20260926 20:59:49.298739 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:49.294981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56063 (local address 127.0.98.1:46009)
0926 20:59:49.295051 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:49.295053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:49.295209 (+ 156us) server_negotiation.cc:407] Connection header received
0926 20:59:49.295309 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:49.295309 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:49.295318 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:49.295344 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:49.295596 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:49.296310 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:49.297823 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:49.298408 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:49.298465 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:49.298475 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:49.298642 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 20:59:49.298679 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":47}
I20260926 20:59:49.298736 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:49.294763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45357 (local address 127.0.98.3:56117)
0926 20:59:49.294931 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:49.294944 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:49.294996 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:49.295339 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:49.295340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:49.295341 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:49.295416 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:49.295418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:49.296435 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:49.296436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:49.297716 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:49.297719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:49.298473 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:49.298474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:49.298544 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:49.298547 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:59:49.298597 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 20:59:49.298646 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":39}
I20260926 20:59:49.298900 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:49.294774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38719 (local address 127.0.98.2:44733)
0926 20:59:49.295142 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:49.295151 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:49.295223 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:49.295426 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:49.295426 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:49.295428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:49.295479 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:49.295481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:49.296484 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:49.296485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:49.297779 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:49.297781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:49.298545 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:49.298546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:49.298610 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:49.298613 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:59:49.298768 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 20:59:49.298801 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":4,"thread_start_us":60,"threads_started":1}
I20260926 20:59:49.298944 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:49.294940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44733 (local address 127.0.98.3:38719)
0926 20:59:49.294979 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:49.294981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:49.295198 (+ 217us) server_negotiation.cc:407] Connection header received
0926 20:59:49.295297 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:49.295298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:49.295305 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:49.295334 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:49.295559 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:49.296330 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:49.297884 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:49.298408 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:49.298473 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:49.298484 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:49.298719 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 20:59:49.298753 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:59:49.299093 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:49.295101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56117 (local address 127.0.98.2:45357)
0926 20:59:49.295197 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:49.295199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:49.295207 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:59:49.295230 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:49.295231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:49.295239 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:49.295290 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:49.295496 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:49.296278 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:49.297816 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:49.298349 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:49.298415 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:49.298427 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:49.298987 (+ 560us) server_negotiation.cc:299] Negotiation successful
0926 20:59:49.299033 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 20:59:49.299172 18325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68e5efdef83464da1579819f928b0d6" candidate_uuid: "cccc12122ec14318a6c51e2ed44cb3dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0659b47504b846339f5a55082a7e113e" is_pre_election: true
I20260926 20:59:49.299247 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:49.294855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46009 (local address 127.0.98.3:56063)
0926 20:59:49.295125 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:49.295137 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:49.295235 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:49.295397 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:49.295397 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:49.295399 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:49.295512 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:49.295514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:49.296438 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:49.296439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:49.297716 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:49.297719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:49.298501 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:49.298501 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:49.298563 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:49.298567 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:59:49.298869 (+ 302us) client_negotiation.cc:240] Negotiation successful
0926 20:59:49.298937 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":70,"thread_start_us":43,"threads_started":1}
I20260926 20:59:49.299322 18325 raft_consensus.cc:2454] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cccc12122ec14318a6c51e2ed44cb3dc in term 0.
I20260926 20:59:49.299302 18194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68e5efdef83464da1579819f928b0d6" candidate_uuid: "0659b47504b846339f5a55082a7e113e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cccc12122ec14318a6c51e2ed44cb3dc" is_pre_election: true
I20260926 20:59:49.299386 18194 raft_consensus.cc:2454] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0659b47504b846339f5a55082a7e113e in term 0.
I20260926 20:59:49.299501 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:49.294763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46009 (local address 127.0.98.2:57867)
0926 20:59:49.294930 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:49.294944 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:49.295032 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:49.295522 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:49.295523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:49.295524 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:49.295580 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:49.295582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:49.296547 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:49.296548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:49.297779 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:49.297781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:49.298575 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:49.298576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:49.298634 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:49.298637 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:59:49.298689 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 20:59:49.298733 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":40,"thread_start_us":73,"threads_started":1}
I20260926 20:59:49.299569 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:49.295097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57867 (local address 127.0.98.1:46009)
0926 20:59:49.295192 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:49.295194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:49.295201 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:59:49.295233 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:49.295234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:49.295245 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:49.295290 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:49.295662 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:49.296472 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:49.297878 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:49.298467 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:49.298527 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:49.298539 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:49.299186 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 20:59:49.299491 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":56,"threads_started":1}
I20260926 20:59:49.299577 18127 leader_election.cc:302] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0659b47504b846339f5a55082a7e113e, cccc12122ec14318a6c51e2ed44cb3dc; no voters:
I20260926 20:59:49.299723 18391 raft_consensus.cc:2781] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:59:49.299782 18391 raft_consensus.cc:482] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:59:49.299808 18391 raft_consensus.cc:3037] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:49.299877 18063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68e5efdef83464da1579819f928b0d6" candidate_uuid: "cccc12122ec14318a6c51e2ed44cb3dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "727d1543b7644ab99894c46ea42ce4b5" is_pre_election: true
I20260926 20:59:49.299880 18062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68e5efdef83464da1579819f928b0d6" candidate_uuid: "0659b47504b846339f5a55082a7e113e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "727d1543b7644ab99894c46ea42ce4b5" is_pre_election: true
I20260926 20:59:49.300083 18063 raft_consensus.cc:2454] T d68e5efdef83464da1579819f928b0d6 P 727d1543b7644ab99894c46ea42ce4b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cccc12122ec14318a6c51e2ed44cb3dc in term 0.
I20260926 20:59:49.300379 18259 leader_election.cc:302] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0659b47504b846339f5a55082a7e113e, cccc12122ec14318a6c51e2ed44cb3dc; no voters: 727d1543b7644ab99894c46ea42ce4b5
I20260926 20:59:49.300594 18393 raft_consensus.cc:2781] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:59:49.300659 18393 raft_consensus.cc:482] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:59:49.300688 18393 raft_consensus.cc:3037] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:49.300907 18391 raft_consensus.cc:504] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cccc12122ec14318a6c51e2ed44cb3dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45357 } } peers { permanent_uuid: "0659b47504b846339f5a55082a7e113e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38719 } } peers { permanent_uuid: "727d1543b7644ab99894c46ea42ce4b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46009 } }
I20260926 20:59:49.301080 18391 leader_election.cc:288] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [CANDIDATE]: Term 1 election: Requested vote from peers 0659b47504b846339f5a55082a7e113e (127.0.98.3:38719), 727d1543b7644ab99894c46ea42ce4b5 (127.0.98.1:46009)
I20260926 20:59:49.301236 18325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68e5efdef83464da1579819f928b0d6" candidate_uuid: "cccc12122ec14318a6c51e2ed44cb3dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0659b47504b846339f5a55082a7e113e"
I20260926 20:59:49.301281 18062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68e5efdef83464da1579819f928b0d6" candidate_uuid: "cccc12122ec14318a6c51e2ed44cb3dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "727d1543b7644ab99894c46ea42ce4b5"
I20260926 20:59:49.301344 18062 raft_consensus.cc:3037] T d68e5efdef83464da1579819f928b0d6 P 727d1543b7644ab99894c46ea42ce4b5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:49.301587 18393 raft_consensus.cc:504] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cccc12122ec14318a6c51e2ed44cb3dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45357 } } peers { permanent_uuid: "0659b47504b846339f5a55082a7e113e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38719 } } peers { permanent_uuid: "727d1543b7644ab99894c46ea42ce4b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46009 } }
I20260926 20:59:49.301733 18393 leader_election.cc:288] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [CANDIDATE]: Term 1 election: Requested vote from peers cccc12122ec14318a6c51e2ed44cb3dc (127.0.98.2:45357), 727d1543b7644ab99894c46ea42ce4b5 (127.0.98.1:46009)
I20260926 20:59:49.301713 18325 raft_consensus.cc:2379] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cccc12122ec14318a6c51e2ed44cb3dc in current term 1: Already voted for candidate 0659b47504b846339f5a55082a7e113e in this term.
I20260926 20:59:49.302011 18063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68e5efdef83464da1579819f928b0d6" candidate_uuid: "0659b47504b846339f5a55082a7e113e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "727d1543b7644ab99894c46ea42ce4b5"
I20260926 20:59:49.302043 18194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68e5efdef83464da1579819f928b0d6" candidate_uuid: "0659b47504b846339f5a55082a7e113e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cccc12122ec14318a6c51e2ed44cb3dc"
I20260926 20:59:49.302155 18194 raft_consensus.cc:2379] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0659b47504b846339f5a55082a7e113e in current term 1: Already voted for candidate cccc12122ec14318a6c51e2ed44cb3dc in this term.
I20260926 20:59:49.302222 18062 raft_consensus.cc:2454] T d68e5efdef83464da1579819f928b0d6 P 727d1543b7644ab99894c46ea42ce4b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cccc12122ec14318a6c51e2ed44cb3dc in term 1.
I20260926 20:59:49.302373 18128 leader_election.cc:302] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 727d1543b7644ab99894c46ea42ce4b5, cccc12122ec14318a6c51e2ed44cb3dc; no voters: 0659b47504b846339f5a55082a7e113e
I20260926 20:59:49.302380 18259 leader_election.cc:302] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0659b47504b846339f5a55082a7e113e; no voters: 727d1543b7644ab99894c46ea42ce4b5, cccc12122ec14318a6c51e2ed44cb3dc
I20260926 20:59:49.302475 18393 raft_consensus.cc:2726] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 20:59:49.302511 18391 raft_consensus.cc:2781] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:59:49.302663 18391 raft_consensus.cc:686] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [term 1 LEADER]: Becoming Leader. State: Replica: cccc12122ec14318a6c51e2ed44cb3dc, State: Running, Role: LEADER
I20260926 20:59:49.302775 18391 consensus_queue.cc:237] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cccc12122ec14318a6c51e2ed44cb3dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45357 } } peers { permanent_uuid: "0659b47504b846339f5a55082a7e113e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38719 } } peers { permanent_uuid: "727d1543b7644ab99894c46ea42ce4b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46009 } }
I20260926 20:59:49.303663 17921 catalog_manager.cc:5495] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc reported cstate change: term changed from 0 to 1, leader changed from <none> to cccc12122ec14318a6c51e2ed44cb3dc (127.0.98.2). New cstate: current_term: 1 leader_uuid: "cccc12122ec14318a6c51e2ed44cb3dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cccc12122ec14318a6c51e2ed44cb3dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0659b47504b846339f5a55082a7e113e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "727d1543b7644ab99894c46ea42ce4b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260926 20:59:49.365526 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:49.362987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.0.98.2:45357)
0926 20:59:49.363047 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:49.363049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:49.363130 (+ 81us) server_negotiation.cc:407] Connection header received
0926 20:59:49.363189 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:49.363190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:49.363208 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:49.363271 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:59:49.363421 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:49.363885 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:49.364643 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:49.364831 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:49.364884 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:49.365077 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:59:49.365213 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:59:49.365357 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:49.365387 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 20:59:49.365410 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:59:49.367398 18325 raft_consensus.cc:1261] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [term 1 FOLLOWER]: Refusing update from remote peer cccc12122ec14318a6c51e2ed44cb3dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:59:49.367443 18062 raft_consensus.cc:1261] T d68e5efdef83464da1579819f928b0d6 P 727d1543b7644ab99894c46ea42ce4b5 [term 1 FOLLOWER]: Refusing update from remote peer cccc12122ec14318a6c51e2ed44cb3dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:59:49.367729 18391 consensus_queue.cc:1035] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0659b47504b846339f5a55082a7e113e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:59:49.367893 18398 consensus_queue.cc:1035] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "727d1543b7644ab99894c46ea42ce4b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:59:49.371232 18405 mvcc.cc:205] Tried to move back new op lower bound from 7333709370846003200 to 7333709370584739840. Current Snapshot: MvccSnapshot[applied={T|T < 7333709370846003200}]
I20260926 20:59:49.373641 17921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:59:49.373731 17921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:59:49.374526 17921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a0fbfe15d5f44b4bb8ed99e18b32b62: Sending DeleteTablet for 3 replicas of tablet d68e5efdef83464da1579819f928b0d6
I20260926 20:59:49.374886 18174 tablet_service.cc:1514] Processing DeleteTablet for tablet d68e5efdef83464da1579819f928b0d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:49 UTC) from {username='slave'} at 127.0.0.1:47920
I20260926 20:59:49.374914 18305 tablet_service.cc:1514] Processing DeleteTablet for tablet d68e5efdef83464da1579819f928b0d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:49 UTC) from {username='slave'} at 127.0.0.1:54126
I20260926 20:59:49.374926 18043 tablet_service.cc:1514] Processing DeleteTablet for tablet d68e5efdef83464da1579819f928b0d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:49 UTC) from {username='slave'} at 127.0.0.1:36906
I20260926 20:59:49.375216 18414 tablet_replica.cc:330] T d68e5efdef83464da1579819f928b0d6 P 727d1543b7644ab99894c46ea42ce4b5: stopping tablet replica
I20260926 20:59:49.375327 18414 raft_consensus.cc:2229] T d68e5efdef83464da1579819f928b0d6 P 727d1543b7644ab99894c46ea42ce4b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:59:49.375344 18413 tablet_replica.cc:330] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e: stopping tablet replica
I20260926 20:59:49.375495 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 17981
I20260926 20:59:49.376224 18413 raft_consensus.cc:2229] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:59:49.376359 18413 raft_consensus.cc:2258] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:59:49.376784 18413 ts_tablet_manager.cc:1905] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:59:49.377193 18412 tablet_replica.cc:330] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc: stopping tablet replica
I20260926 20:59:49.377282 18412 raft_consensus.cc:2229] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:59:49.377391 18412 raft_consensus.cc:2258] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:59:49.377817 18412 ts_tablet_manager.cc:1905] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:59:49.378814 18413 ts_tablet_manager.cc:1918] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:59:49.378885 18413 log.cc:1196] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/d68e5efdef83464da1579819f928b0d6
I20260926 20:59:49.379135 18413 ts_tablet_manager.cc:1939] T d68e5efdef83464da1579819f928b0d6 P 0659b47504b846339f5a55082a7e113e: Deleting consensus metadata
I20260926 20:59:49.379599 18412 ts_tablet_manager.cc:1918] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:59:49.379616 17905 catalog_manager.cc:4841] TS 0659b47504b846339f5a55082a7e113e (127.0.98.3:38719): tablet d68e5efdef83464da1579819f928b0d6 (table test-table [id=8a32f78448f945a8b6ad4856c96942e2]) successfully deleted
I20260926 20:59:49.379662 18412 log.cc:1196] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/d68e5efdef83464da1579819f928b0d6
I20260926 20:59:49.379918 18412 ts_tablet_manager.cc:1939] T d68e5efdef83464da1579819f928b0d6 P cccc12122ec14318a6c51e2ed44cb3dc: Deleting consensus metadata
W20260926 20:59:49.381817 17906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46009 (error 108)
I20260926 20:59:49.382299 17906 catalog_manager.cc:4841] TS cccc12122ec14318a6c51e2ed44cb3dc (127.0.98.2:45357): tablet d68e5efdef83464da1579819f928b0d6 (table test-table [id=8a32f78448f945a8b6ad4856c96942e2]) successfully deleted
W20260926 20:59:49.382400 18416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:59:49.382210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46009 (local address 127.0.0.1:36912)
0926 20:59:49.382301 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:49.382347 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46009: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":50,"threads_started":1}
W20260926 20:59:49.382573 17906 catalog_manager.cc:4584] TS 727d1543b7644ab99894c46ea42ce4b5 (127.0.98.1:46009): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d68e5efdef83464da1579819f928b0d6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46009: connect: Connection reset by peer (error 104)
I20260926 20:59:49.382647 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 18112
I20260926 20:59:49.386981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 18243
I20260926 20:59:49.391335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 17890
2026-09-26T20:59:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1149 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:59:49 dist-test-slave-fl7g krb5kdc[18419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:59:49 dist-test-slave-fl7g krb5kdc[18419](info): set up 2 sockets
Sep 26 20:59:49 dist-test-slave-fl7g krb5kdc[18419](info): commencing operation
krb5kdc: starting...
W20260926 20:59:50.424227 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:59:50 dist-test-slave-fl7g krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:59:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:59:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:59:50.548666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32869
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:50.645988 18432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:50.646236 18432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:50.646263 18432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:50.647557 18432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:59:50.647598 18432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:50.647612 18432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:50.647624 18432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:59:50.647637 18432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:59:50.649189 18432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:50.649410 18432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:50.649602 18432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:50.654722 18437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:50.654731 18440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:50.654876 18432 server_base.cc:1034] running on GCE node
W20260926 20:59:50.654722 18438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:50.655349 18432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:50.655684 18432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:50.656849 18432 hybrid_clock.cc:648] HybridClock initialized: now 1790456390656839 us; error 53 us; skew 500 ppm
Sep 26 20:59:50 dist-test-slave-fl7g krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:50.659435 18432 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:59:50.659919 18432 webserver.cc:460] Webserver started at http://127.0.98.62:45733/ using document root <none> and password file <none>
I20260926 20:59:50.660157 18432 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:50.660226 18432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:50.660382 18432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:50.661520 18432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "64187354f7e8488da5941336382dcf68"
format_stamp: "Formatted at 2026-09-26 20:59:50 on dist-test-slave-fl7g"
server_key: "4b82a4d1c95dd3ff2a8acc12dbce0444"
server_key_iv: "b4f0b69fdb6da402e86da3a9583852f2"
server_key_version: "encryptionkey@0"
I20260926 20:59:50.661919 18432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "64187354f7e8488da5941336382dcf68"
format_stamp: "Formatted at 2026-09-26 20:59:50 on dist-test-slave-fl7g"
server_key: "4b82a4d1c95dd3ff2a8acc12dbce0444"
server_key_iv: "b4f0b69fdb6da402e86da3a9583852f2"
server_key_version: "encryptionkey@0"
I20260926 20:59:50.663307 18432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:59:50.664018 18447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:50.664197 18432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:59:50.664301 18432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "64187354f7e8488da5941336382dcf68"
format_stamp: "Formatted at 2026-09-26 20:59:50 on dist-test-slave-fl7g"
server_key: "4b82a4d1c95dd3ff2a8acc12dbce0444"
server_key_iv: "b4f0b69fdb6da402e86da3a9583852f2"
server_key_version: "encryptionkey@0"
I20260926 20:59:50.664387 18432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:50.678274 18432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:50.680150 18432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:50.680330 18432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:50.684307 18432 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32869
I20260926 20:59:50.684345 18499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32869 every 8 connection(s)
I20260926 20:59:50.684741 18432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:59:50.685689 18500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:50.688079 18500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68: Bootstrap starting.
I20260926 20:59:50.688755 18500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:50.689057 18500 log.cc:824] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:50.689823 18500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68: No bootstrap required, opened a new log
I20260926 20:59:50.691452 18500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64187354f7e8488da5941336382dcf68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32869 } }
I20260926 20:59:50.691596 18500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:50.691618 18500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64187354f7e8488da5941336382dcf68, State: Initialized, Role: FOLLOWER
I20260926 20:59:50.691795 18500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64187354f7e8488da5941336382dcf68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32869 } }
I20260926 20:59:50.691864 18500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:59:50.691884 18500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:59:50.692058 18500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:50.692793 18500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64187354f7e8488da5941336382dcf68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32869 } }
I20260926 20:59:50.692910 18500 leader_election.cc:302] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64187354f7e8488da5941336382dcf68; no voters:
I20260926 20:59:50.693112 18500 leader_election.cc:288] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:59:50.693151 18503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:59:50.693336 18503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68 [term 1 LEADER]: Becoming Leader. State: Replica: 64187354f7e8488da5941336382dcf68, State: Running, Role: LEADER
I20260926 20:59:50.693379 18500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:59:50.693432 18503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64187354f7e8488da5941336382dcf68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32869 } }
I20260926 20:59:50.693867 18505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64187354f7e8488da5941336382dcf68. Latest consensus state: current_term: 1 leader_uuid: "64187354f7e8488da5941336382dcf68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64187354f7e8488da5941336382dcf68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32869 } } }
I20260926 20:59:50.693951 18505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68 [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:50.694267 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 18432
I20260926 20:59:50.694371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:59:50.694399 18512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:59:50.694614 392 external_mini_cluster.cc:1428] Setting key 61a88efbe377f9d500a0e638f1e42e6e
I20260926 20:59:50.694715 18504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64187354f7e8488da5941336382dcf68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64187354f7e8488da5941336382dcf68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32869 } } }
I20260926 20:59:50.694821 18504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68 [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:50.694919 18512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:59:50.696877 18512 catalog_manager.cc:1347] Generated new cluster ID: 630a464186e242b99f0f835643622bf1
I20260926 20:59:50.696944 18512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:59:50 dist-test-slave-fl7g krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:59:50.708535 18512 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:59:50.709224 18512 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:59:50.710723 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:50.695245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46842 (local address 127.0.98.62:32869)
0926 20:59:50.695442 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:50.695451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:50.695483 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:59:50.695772 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:50.695780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:50.695841 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:50.695992 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:50.700553 (+ 4561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:50.702422 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:50.703478 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:50.704112 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:50.704254 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:50.707222 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:50.707226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:50.707227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:50.707240 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:50.709134 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:50.709536 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:50.709539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:50.709539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:50.709638 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:50.709854 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:50.709855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:50.709855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:50.710081 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:50.710224 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:50.710330 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 20:59:50.710487 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260926 20:59:50.713068 18512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68: Generated new TSK 0
I20260926 20:59:50.713270 18512 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:59:50.739753 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32869
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:50.824369 18526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:50.824651 18526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:50.824692 18526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:50.825958 18526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:50.826026 18526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:50.826073 18526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:59:50.827629 18526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:50.827931 18526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:50.828172 18526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:50.833297 18534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:50.833308 18531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:50.833513 18526 server_base.cc:1034] running on GCE node
W20260926 20:59:50.833298 18532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:50.833809 18526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:50.834009 18526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:50.835157 18526 hybrid_clock.cc:648] HybridClock initialized: now 1790456390835144 us; error 28 us; skew 500 ppm
Sep 26 20:59:50 dist-test-slave-fl7g krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:50.837867 18526 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:59:50.838368 18526 webserver.cc:460] Webserver started at http://127.0.98.1:42847/ using document root <none> and password file <none>
I20260926 20:59:50.838609 18526 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:50.838680 18526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:50.838837 18526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:50.839931 18526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "187ea4ac3cf744a6af66159f04cfc837"
format_stamp: "Formatted at 2026-09-26 20:59:50 on dist-test-slave-fl7g"
server_key: "489ffe17692dcc7459f24db0a0c4ca38"
server_key_iv: "dec9359ef0d32b9d4c8a9830ab4099c3"
server_key_version: "encryptionkey@0"
I20260926 20:59:50.840304 18526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "187ea4ac3cf744a6af66159f04cfc837"
format_stamp: "Formatted at 2026-09-26 20:59:50 on dist-test-slave-fl7g"
server_key: "489ffe17692dcc7459f24db0a0c4ca38"
server_key_iv: "dec9359ef0d32b9d4c8a9830ab4099c3"
server_key_version: "encryptionkey@0"
I20260926 20:59:50.841722 18526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:59:50.842473 18541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:50.842653 18526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:59:50.842777 18526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "187ea4ac3cf744a6af66159f04cfc837"
format_stamp: "Formatted at 2026-09-26 20:59:50 on dist-test-slave-fl7g"
server_key: "489ffe17692dcc7459f24db0a0c4ca38"
server_key_iv: "dec9359ef0d32b9d4c8a9830ab4099c3"
server_key_version: "encryptionkey@0"
I20260926 20:59:50.842864 18526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:50.863695 18526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:50.865832 18526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:50.866001 18526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:50.866248 18526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:50.866571 18526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:50.866626 18526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:50.866670 18526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:50.866709 18526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:50.873370 18526 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46745
I20260926 20:59:50.873430 18654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46745 every 8 connection(s)
I20260926 20:59:50.873809 18526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:59:50.875383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 18526
I20260926 20:59:50.875473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:59:50.875715 392 external_mini_cluster.cc:1428] Setting key 62b5d43d4307e65e73d8679a8aeee012
Sep 26 20:59:50 dist-test-slave-fl7g krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 20:59:50.884508 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:50.874626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33303 (local address 127.0.98.62:32869)
0926 20:59:50.874692 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:50.874694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:50.874940 (+ 246us) server_negotiation.cc:407] Connection header received
0926 20:59:50.876102 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:50.876104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:50.876116 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:50.876174 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:50.877018 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:50.877544 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:50.878603 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:50.878831 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:50.878924 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:50.881232 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:50.881233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:50.881234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:50.881243 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:50.882906 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:50.883363 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:50.883364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:50.883364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:50.883432 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:50.883727 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:50.883728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:50.883729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:50.883862 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:50.883926 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:50.884242 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 20:59:50.884359 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:59:50.884811 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:50.874592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32869 (local address 127.0.98.1:33303)
0926 20:59:50.874811 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:50.874854 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:50.876007 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:50.876246 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:50.876250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:50.876711 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:50.876918 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:50.876923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:50.877657 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:50.877658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:50.878493 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:50.878496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:50.878935 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:50.878936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:50.879087 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:50.879662 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:59:50.879671 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:50.881129 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:50.883024 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:50.883032 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:50.883035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:50.883279 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:50.883522 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:50.883523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:50.883523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:50.883645 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:50.883956 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:50.883963 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:50.884105 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 20:59:50.884215 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 20:59:50.884471 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":121,"threads_started":1}
I20260926 20:59:50.885413 18655 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32869
I20260926 20:59:50.885516 18655 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:50.885670 18655 heartbeater.cc:511] Master 127.0.98.62:32869 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 20:59:50.886232 18464 ts_manager.cc:194] Registered new tserver with Master: 187ea4ac3cf744a6af66159f04cfc837 (127.0.98.1:46745)
I20260926 20:59:50.886864 18464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:59:50.902957 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32869
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:50.988351 18660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:50.988591 18660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:50.988617 18660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:50.989847 18660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:50.989893 18660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:50.989921 18660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:59:50.991460 18660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:50.991730 18660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:50.991977 18660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:50.997535 18665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:50.997500 18668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:50.997568 18666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:50.997759 18660 server_base.cc:1034] running on GCE node
I20260926 20:59:50.997987 18660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:50.998195 18660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:50.999331 18660 hybrid_clock.cc:648] HybridClock initialized: now 1790456390999326 us; error 33 us; skew 500 ppm
Sep 26 20:59:51 dist-test-slave-fl7g krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:51.001914 18660 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:59:51.002311 18660 webserver.cc:460] Webserver started at http://127.0.98.2:35667/ using document root <none> and password file <none>
I20260926 20:59:51.002507 18660 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:51.002547 18660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:51.002633 18660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:51.003733 18660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "34d75d07a15645518bb3abb82fb4ea1e"
format_stamp: "Formatted at 2026-09-26 20:59:51 on dist-test-slave-fl7g"
server_key: "b5e16865382ce514b705b66051e2bd43"
server_key_iv: "fca66359ee5e37c8b6fe4e1988227635"
server_key_version: "encryptionkey@0"
I20260926 20:59:51.004074 18660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "34d75d07a15645518bb3abb82fb4ea1e"
format_stamp: "Formatted at 2026-09-26 20:59:51 on dist-test-slave-fl7g"
server_key: "b5e16865382ce514b705b66051e2bd43"
server_key_iv: "fca66359ee5e37c8b6fe4e1988227635"
server_key_version: "encryptionkey@0"
I20260926 20:59:51.005486 18660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:59:51.006273 18675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:51.006491 18660 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:51.006567 18660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "34d75d07a15645518bb3abb82fb4ea1e"
format_stamp: "Formatted at 2026-09-26 20:59:51 on dist-test-slave-fl7g"
server_key: "b5e16865382ce514b705b66051e2bd43"
server_key_iv: "fca66359ee5e37c8b6fe4e1988227635"
server_key_version: "encryptionkey@0"
I20260926 20:59:51.006664 18660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:51.021162 18660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:51.023183 18660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:51.023370 18660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:51.023627 18660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:51.023968 18660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:51.024024 18660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:51.024067 18660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:51.024096 18660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:51.030278 18660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37203
I20260926 20:59:51.030377 18788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37203 every 8 connection(s)
I20260926 20:59:51.030726 18660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:59:51 dist-test-slave-fl7g krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:59:51.038434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 18660
I20260926 20:59:51.038568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:59:51.038761 392 external_mini_cluster.cc:1428] Setting key 9fcb424f1206cf3e9d2f9c4a7bc89769
I20260926 20:59:51.041253 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.031533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35123 (local address 127.0.98.62:32869)
0926 20:59:51.031594 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:51.031595 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:51.031825 (+ 230us) server_negotiation.cc:407] Connection header received
0926 20:59:51.032758 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:51.032759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:51.032769 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:51.032827 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:51.033434 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.033926 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.034992 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.035301 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.035397 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.037867 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:51.037868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:51.037868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:51.037876 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:51.039587 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:51.040100 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:51.040101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:51.040101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:51.040152 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:51.040466 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:51.040467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:51.040467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:51.040584 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:51.040637 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:51.040996 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 20:59:51.041127 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:59:51.041440 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.031507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32869 (local address 127.0.98.2:35123)
0926 20:59:51.031656 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:51.031703 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:51.032657 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:51.032890 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:51.032893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:51.033157 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:51.033352 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:51.033357 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.034027 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.034027 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:51.034883 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:51.034886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.035406 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.035407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:51.035586 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.036218 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:59:51.036226 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:51.037752 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:51.039664 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:51.039673 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:51.039675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:51.040017 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:51.040211 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:51.040212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:51.040212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:51.040376 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:51.040652 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:51.040660 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:51.040847 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 20:59:51.040959 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 20:59:51.041174 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260926 20:59:51.041921 18789 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32869
I20260926 20:59:51.042033 18789 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:51.042253 18789 heartbeater.cc:511] Master 127.0.98.62:32869 requested a full tablet report, sending...
I20260926 20:59:51.042627 18464 ts_manager.cc:194] Registered new tserver with Master: 34d75d07a15645518bb3abb82fb4ea1e (127.0.98.2:37203)
I20260926 20:59:51.043063 18464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35123
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:59:51.067420 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32869
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:51.149683 18794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:51.149955 18794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:51.149998 18794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:51.151203 18794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:51.151283 18794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:51.151327 18794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:59:51.153016 18794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:51.153301 18794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:51.153546 18794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:51.159157 18800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:51.159147 18799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:51.159196 18794 server_base.cc:1034] running on GCE node
W20260926 20:59:51.159130 18802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:51.159631 18794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:51.159905 18794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:51.161067 18794 hybrid_clock.cc:648] HybridClock initialized: now 1790456391161059 us; error 47 us; skew 500 ppm
Sep 26 20:59:51 dist-test-slave-fl7g krb5kdc[18419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:51.163614 18794 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:59:51.164220 18794 webserver.cc:460] Webserver started at http://127.0.98.3:43393/ using document root <none> and password file <none>
I20260926 20:59:51.164541 18794 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:51.164609 18794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:51.164747 18794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:51.165880 18794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "718d70a5aef745a8bd1ac21cb473cbba"
format_stamp: "Formatted at 2026-09-26 20:59:51 on dist-test-slave-fl7g"
server_key: "e88ec2eefedd409daf346a1c2a85eb25"
server_key_iv: "362d70c3e61eb8b7c7750969064c6efe"
server_key_version: "encryptionkey@0"
I20260926 20:59:51.166254 18794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "718d70a5aef745a8bd1ac21cb473cbba"
format_stamp: "Formatted at 2026-09-26 20:59:51 on dist-test-slave-fl7g"
server_key: "e88ec2eefedd409daf346a1c2a85eb25"
server_key_iv: "362d70c3e61eb8b7c7750969064c6efe"
server_key_version: "encryptionkey@0"
I20260926 20:59:51.167689 18794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:59:51.168542 18809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:51.168735 18794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:59:51.168852 18794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "718d70a5aef745a8bd1ac21cb473cbba"
format_stamp: "Formatted at 2026-09-26 20:59:51 on dist-test-slave-fl7g"
server_key: "e88ec2eefedd409daf346a1c2a85eb25"
server_key_iv: "362d70c3e61eb8b7c7750969064c6efe"
server_key_version: "encryptionkey@0"
I20260926 20:59:51.168942 18794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:51.203681 18794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:51.206211 18794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:51.206387 18794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:51.206638 18794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:51.206961 18794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:51.207016 18794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:51.207060 18794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:51.207088 18794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:51.213368 18794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39031
I20260926 20:59:51.213409 18922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39031 every 8 connection(s)
I20260926 20:59:51.213788 18794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:59:51 dist-test-slave-fl7g krb5kdc[18419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:59:51.223040 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 18794
I20260926 20:59:51.223150 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:59:51.223357 392 external_mini_cluster.cc:1428] Setting key c2a4e8c4d4f76ab7851e403600afc10f
I20260926 20:59:51.223363 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.214582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47379 (local address 127.0.98.62:32869)
0926 20:59:51.214645 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:51.214648 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:51.214826 (+ 178us) server_negotiation.cc:407] Connection header received
0926 20:59:51.215828 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:51.215829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:51.215843 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:51.215896 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:51.216600 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.217114 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.218148 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.218352 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.218438 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.220538 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:51.220540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:51.220540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:51.220548 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:51.221875 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:51.222314 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:51.222316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:51.222316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:51.222380 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:51.222667 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:51.222670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:51.222670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:51.222781 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:51.222854 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:51.223213 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 20:59:51.223297 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 20:59:51.223676 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.214538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32869 (local address 127.0.98.3:47379)
0926 20:59:51.214700 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:51.214737 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:51.215689 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:51.215985 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:51.215989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:51.216253 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:51.216511 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:51.216515 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.217219 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.217220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:51.218057 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:51.218060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.218448 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.218449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:51.218578 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.219050 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:59:51.219058 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:51.220396 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:51.221987 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:51.221994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:51.221996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:51.222232 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:51.222447 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:51.222448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:51.222448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:51.222563 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:51.222857 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:51.222864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:51.223036 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 20:59:51.223174 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 20:59:51.223351 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260926 20:59:51.224143 18923 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32869
I20260926 20:59:51.224233 18923 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:51.224459 18923 heartbeater.cc:511] Master 127.0.98.62:32869 requested a full tablet report, sending...
I20260926 20:59:51.224820 18464 ts_manager.cc:194] Registered new tserver with Master: 718d70a5aef745a8bd1ac21cb473cbba (127.0.98.3:39031)
I20260926 20:59:51.225154 18464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47379
I20260926 20:59:51.235824 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:59:51.248986 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.237459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.0.98.62:32869)
0926 20:59:51.237554 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:51.237557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:51.237680 (+ 123us) server_negotiation.cc:407] Connection header received
0926 20:59:51.237775 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:51.237776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:51.237788 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:51.237863 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:51.238470 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.238971 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.239735 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.239936 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.240022 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.246300 (+ 6278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:51.246302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:51.246302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:51.246312 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:51.247580 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:51.248020 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:51.248021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:51.248022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:51.248083 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:51.248319 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:51.248320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:51.248320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:51.248535 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:51.248657 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:51.248804 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 20:59:51.248874 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:59:51.250576 18464 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:59:51.252329 18464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:59:51.259051 18937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.254995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55118 (local address 127.0.98.3:39031)
0926 20:59:51.255247 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:51.255253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:51.255265 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:59:51.255308 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:51.255311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:51.255328 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:51.255424 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:51.255634 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.256694 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.257877 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.258592 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.258655 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.258710 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:51.258914 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 20:59:51.258956 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":83,"threads_started":1}
I20260926 20:59:51.259037 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.254799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39031 (local address 127.0.0.1:55118)
0926 20:59:51.254978 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:51.255021 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:51.255086 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:51.255436 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:51.255439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:51.255441 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:51.255531 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:51.255533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.256795 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.256796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:51.257765 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:51.257767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.258737 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.258738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:51.258795 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.258813 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 20:59:51.258899 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 20:59:51.258955 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":66,"threads_started":1}
I20260926 20:59:51.259374 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.254829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37203 (local address 127.0.0.1:56666)
0926 20:59:51.255517 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:51.255526 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:51.255562 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:51.255865 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:51.255866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:51.255869 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:51.255914 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:51.255916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.257278 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.257279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:51.258291 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:51.258294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.259149 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.259150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:51.259214 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.259220 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:59:51.259252 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 20:59:51.259293 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":66,"thread_start_us":77,"threads_started":1}
I20260926 20:59:51.259754 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.255009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46745 (local address 127.0.0.1:60806)
0926 20:59:51.255972 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:51.255982 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:51.256033 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:51.256205 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:51.256206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:51.256208 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:51.256249 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:51.256251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.257465 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.257466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:51.258536 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:51.258539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.259535 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.259536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:51.259595 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.259598 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:59:51.259656 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:59:51.259693 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"thread_start_us":76,"threads_started":1}
I20260926 20:59:51.259779 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.255209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60806 (local address 127.0.98.1:46745)
0926 20:59:51.255381 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:51.255405 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:51.256032 (+ 627us) server_negotiation.cc:407] Connection header received
0926 20:59:51.256099 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:51.256101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:51.256117 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:51.256192 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:51.256309 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.257359 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.258630 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.259373 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.259441 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.259499 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:51.259683 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 20:59:51.259721 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260926 20:59:51.259975 18857 tablet_service.cc:1467] Processing CreateTablet for tablet 9e05207caf894f12a5e0a9f1f08363b4 (DEFAULT_TABLE table=test-table [id=54eb6b1508e74cedb2dfabc1c01cc819]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:51.260375 18857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e05207caf894f12a5e0a9f1f08363b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:51.261202 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 9e05207caf894f12a5e0a9f1f08363b4 (DEFAULT_TABLE table=test-table [id=54eb6b1508e74cedb2dfabc1c01cc819]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:51.261551 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e05207caf894f12a5e0a9f1f08363b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:51.264024 18940 tablet_bootstrap.cc:492] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba: Bootstrap starting.
I20260926 20:59:51.264101 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.254926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.0.98.2:37203)
0926 20:59:51.255716 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:51.255720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:51.255729 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:59:51.255765 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:51.255767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:51.255782 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:51.255867 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:51.256002 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.257161 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.258379 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.259047 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.263806 (+ 4759us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.263914 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:51.263948 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 20:59:51.264008 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":91,"threads_started":1}
I20260926 20:59:51.264906 18940 tablet_bootstrap.cc:654] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:51.264886 18723 tablet_service.cc:1467] Processing CreateTablet for tablet 9e05207caf894f12a5e0a9f1f08363b4 (DEFAULT_TABLE table=test-table [id=54eb6b1508e74cedb2dfabc1c01cc819]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:51.265020 18941 tablet_bootstrap.cc:492] T 9e05207caf894f12a5e0a9f1f08363b4 P 187ea4ac3cf744a6af66159f04cfc837: Bootstrap starting.
I20260926 20:59:51.265228 18723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e05207caf894f12a5e0a9f1f08363b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:51.265703 18941 tablet_bootstrap.cc:654] T 9e05207caf894f12a5e0a9f1f08363b4 P 187ea4ac3cf744a6af66159f04cfc837: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:51.265686 18940 log.cc:824] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:51.266238 18941 log.cc:824] T 9e05207caf894f12a5e0a9f1f08363b4 P 187ea4ac3cf744a6af66159f04cfc837: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:51.266520 18940 tablet_bootstrap.cc:492] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba: No bootstrap required, opened a new log
I20260926 20:59:51.266618 18940 ts_tablet_manager.cc:1397] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:59:51.267057 18941 tablet_bootstrap.cc:492] T 9e05207caf894f12a5e0a9f1f08363b4 P 187ea4ac3cf744a6af66159f04cfc837: No bootstrap required, opened a new log
I20260926 20:59:51.267189 18941 ts_tablet_manager.cc:1397] T 9e05207caf894f12a5e0a9f1f08363b4 P 187ea4ac3cf744a6af66159f04cfc837: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:59:51.268460 18944 tablet_bootstrap.cc:492] T 9e05207caf894f12a5e0a9f1f08363b4 P 34d75d07a15645518bb3abb82fb4ea1e: Bootstrap starting.
I20260926 20:59:51.268985 18944 tablet_bootstrap.cc:654] T 9e05207caf894f12a5e0a9f1f08363b4 P 34d75d07a15645518bb3abb82fb4ea1e: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:51.268901 18940 raft_consensus.cc:348] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d75d07a15645518bb3abb82fb4ea1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37203 } } peers { permanent_uuid: "187ea4ac3cf744a6af66159f04cfc837" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46745 } } peers { permanent_uuid: "718d70a5aef745a8bd1ac21cb473cbba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39031 } }
I20260926 20:59:51.269076 18940 raft_consensus.cc:374] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:51.269119 18940 raft_consensus.cc:729] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718d70a5aef745a8bd1ac21cb473cbba, State: Initialized, Role: FOLLOWER
I20260926 20:59:51.269261 18940 consensus_queue.cc:260] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d75d07a15645518bb3abb82fb4ea1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37203 } } peers { permanent_uuid: "187ea4ac3cf744a6af66159f04cfc837" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46745 } } peers { permanent_uuid: "718d70a5aef745a8bd1ac21cb473cbba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39031 } }
I20260926 20:59:51.269282 18941 raft_consensus.cc:348] T 9e05207caf894f12a5e0a9f1f08363b4 P 187ea4ac3cf744a6af66159f04cfc837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d75d07a15645518bb3abb82fb4ea1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37203 } } peers { permanent_uuid: "187ea4ac3cf744a6af66159f04cfc837" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46745 } } peers { permanent_uuid: "718d70a5aef745a8bd1ac21cb473cbba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39031 } }
I20260926 20:59:51.269469 18941 raft_consensus.cc:374] T 9e05207caf894f12a5e0a9f1f08363b4 P 187ea4ac3cf744a6af66159f04cfc837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:51.269515 18941 raft_consensus.cc:729] T 9e05207caf894f12a5e0a9f1f08363b4 P 187ea4ac3cf744a6af66159f04cfc837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 187ea4ac3cf744a6af66159f04cfc837, State: Initialized, Role: FOLLOWER
I20260926 20:59:51.269555 18940 ts_tablet_manager.cc:1428] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:59:51.269572 18944 log.cc:824] T 9e05207caf894f12a5e0a9f1f08363b4 P 34d75d07a15645518bb3abb82fb4ea1e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:51.269620 18941 consensus_queue.cc:260] T 9e05207caf894f12a5e0a9f1f08363b4 P 187ea4ac3cf744a6af66159f04cfc837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d75d07a15645518bb3abb82fb4ea1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37203 } } peers { permanent_uuid: "187ea4ac3cf744a6af66159f04cfc837" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46745 } } peers { permanent_uuid: "718d70a5aef745a8bd1ac21cb473cbba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39031 } }
I20260926 20:59:51.269670 18923 heartbeater.cc:503] Master 127.0.98.62:32869 was elected leader, sending a full tablet report...
I20260926 20:59:51.269883 18941 ts_tablet_manager.cc:1428] T 9e05207caf894f12a5e0a9f1f08363b4 P 187ea4ac3cf744a6af66159f04cfc837: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:59:51.270001 18655 heartbeater.cc:503] Master 127.0.98.62:32869 was elected leader, sending a full tablet report...
I20260926 20:59:51.270364 18944 tablet_bootstrap.cc:492] T 9e05207caf894f12a5e0a9f1f08363b4 P 34d75d07a15645518bb3abb82fb4ea1e: No bootstrap required, opened a new log
I20260926 20:59:51.270442 18944 ts_tablet_manager.cc:1397] T 9e05207caf894f12a5e0a9f1f08363b4 P 34d75d07a15645518bb3abb82fb4ea1e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:59:51.272264 18944 raft_consensus.cc:348] T 9e05207caf894f12a5e0a9f1f08363b4 P 34d75d07a15645518bb3abb82fb4ea1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d75d07a15645518bb3abb82fb4ea1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37203 } } peers { permanent_uuid: "187ea4ac3cf744a6af66159f04cfc837" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46745 } } peers { permanent_uuid: "718d70a5aef745a8bd1ac21cb473cbba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39031 } }
I20260926 20:59:51.272469 18944 raft_consensus.cc:374] T 9e05207caf894f12a5e0a9f1f08363b4 P 34d75d07a15645518bb3abb82fb4ea1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:51.272516 18944 raft_consensus.cc:729] T 9e05207caf894f12a5e0a9f1f08363b4 P 34d75d07a15645518bb3abb82fb4ea1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d75d07a15645518bb3abb82fb4ea1e, State: Initialized, Role: FOLLOWER
I20260926 20:59:51.272635 18944 consensus_queue.cc:260] T 9e05207caf894f12a5e0a9f1f08363b4 P 34d75d07a15645518bb3abb82fb4ea1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d75d07a15645518bb3abb82fb4ea1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37203 } } peers { permanent_uuid: "187ea4ac3cf744a6af66159f04cfc837" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46745 } } peers { permanent_uuid: "718d70a5aef745a8bd1ac21cb473cbba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39031 } }
I20260926 20:59:51.272874 18944 ts_tablet_manager.cc:1428] T 9e05207caf894f12a5e0a9f1f08363b4 P 34d75d07a15645518bb3abb82fb4ea1e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:59:51.272935 18789 heartbeater.cc:503] Master 127.0.98.62:32869 was elected leader, sending a full tablet report...
W20260926 20:59:51.281347 18790 tablet.cc:2369] T 9e05207caf894f12a5e0a9f1f08363b4 P 34d75d07a15645518bb3abb82fb4ea1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:59:51.374648 18656 tablet.cc:2369] T 9e05207caf894f12a5e0a9f1f08363b4 P 187ea4ac3cf744a6af66159f04cfc837: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:59:51.410972 18945 raft_consensus.cc:482] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:59:51.411180 18945 raft_consensus.cc:504] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d75d07a15645518bb3abb82fb4ea1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37203 } } peers { permanent_uuid: "187ea4ac3cf744a6af66159f04cfc837" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46745 } } peers { permanent_uuid: "718d70a5aef745a8bd1ac21cb473cbba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39031 } }
I20260926 20:59:51.411568 18945 leader_election.cc:288] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34d75d07a15645518bb3abb82fb4ea1e (127.0.98.2:37203), 187ea4ac3cf744a6af66159f04cfc837 (127.0.98.1:46745)
I20260926 20:59:51.414745 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.412034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59195 (local address 127.0.98.1:46745)
0926 20:59:51.412078 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:51.412080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:51.412235 (+ 155us) server_negotiation.cc:407] Connection header received
0926 20:59:51.412275 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:51.412276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:51.412289 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:51.412358 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:51.412551 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.413038 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.414004 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.414359 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.414451 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.414461 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:51.414625 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 20:59:51.414668 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:59:51.414731 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.411828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37203 (local address 127.0.98.3:41449)
0926 20:59:51.411921 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:51.411938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:51.412008 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:51.412190 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:51.412191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:51.412193 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:51.412289 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:51.412291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.413022 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.413023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:51.413829 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:51.413831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.414425 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.414427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:51.414510 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.414513 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:59:51.414566 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 20:59:51.414632 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 20:59:51.414726 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.411961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41449 (local address 127.0.98.2:37203)
0926 20:59:51.412012 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:51.412014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:51.412024 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:59:51.412096 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:51.412097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:51.412113 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:51.412168 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:51.412386 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.412935 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.413930 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.414339 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.414394 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.414403 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:51.414598 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 20:59:51.414644 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 20:59:51.414736 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.411960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46745 (local address 127.0.98.3:59195)
0926 20:59:51.412154 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:51.412164 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:51.412223 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:51.412374 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:51.412375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:51.412377 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:51.412473 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:51.412475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.413118 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.413119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:51.413912 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:51.413915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.414474 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.414475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:51.414529 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.414532 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:59:51.414607 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:59:51.414644 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":102,"threads_started":1}
I20260926 20:59:51.415352 18609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e05207caf894f12a5e0a9f1f08363b4" candidate_uuid: "718d70a5aef745a8bd1ac21cb473cbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "187ea4ac3cf744a6af66159f04cfc837" is_pre_election: true
I20260926 20:59:51.415354 18743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e05207caf894f12a5e0a9f1f08363b4" candidate_uuid: "718d70a5aef745a8bd1ac21cb473cbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d75d07a15645518bb3abb82fb4ea1e" is_pre_election: true
I20260926 20:59:51.415508 18743 raft_consensus.cc:2454] T 9e05207caf894f12a5e0a9f1f08363b4 P 34d75d07a15645518bb3abb82fb4ea1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 718d70a5aef745a8bd1ac21cb473cbba in term 0.
I20260926 20:59:51.415517 18609 raft_consensus.cc:2454] T 9e05207caf894f12a5e0a9f1f08363b4 P 187ea4ac3cf744a6af66159f04cfc837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 718d70a5aef745a8bd1ac21cb473cbba in term 0.
I20260926 20:59:51.415741 18810 leader_election.cc:302] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 187ea4ac3cf744a6af66159f04cfc837, 718d70a5aef745a8bd1ac21cb473cbba; no voters:
I20260926 20:59:51.415911 18945 raft_consensus.cc:2781] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:59:51.415983 18945 raft_consensus.cc:482] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:59:51.416019 18945 raft_consensus.cc:3037] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:51.416826 18945 raft_consensus.cc:504] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d75d07a15645518bb3abb82fb4ea1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37203 } } peers { permanent_uuid: "187ea4ac3cf744a6af66159f04cfc837" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46745 } } peers { permanent_uuid: "718d70a5aef745a8bd1ac21cb473cbba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39031 } }
I20260926 20:59:51.416983 18945 leader_election.cc:288] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [CANDIDATE]: Term 1 election: Requested vote from peers 34d75d07a15645518bb3abb82fb4ea1e (127.0.98.2:37203), 187ea4ac3cf744a6af66159f04cfc837 (127.0.98.1:46745)
I20260926 20:59:51.417171 18743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e05207caf894f12a5e0a9f1f08363b4" candidate_uuid: "718d70a5aef745a8bd1ac21cb473cbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d75d07a15645518bb3abb82fb4ea1e"
I20260926 20:59:51.417230 18609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e05207caf894f12a5e0a9f1f08363b4" candidate_uuid: "718d70a5aef745a8bd1ac21cb473cbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "187ea4ac3cf744a6af66159f04cfc837"
I20260926 20:59:51.417272 18743 raft_consensus.cc:3037] T 9e05207caf894f12a5e0a9f1f08363b4 P 34d75d07a15645518bb3abb82fb4ea1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:51.417330 18609 raft_consensus.cc:3037] T 9e05207caf894f12a5e0a9f1f08363b4 P 187ea4ac3cf744a6af66159f04cfc837 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:51.418009 18743 raft_consensus.cc:2454] T 9e05207caf894f12a5e0a9f1f08363b4 P 34d75d07a15645518bb3abb82fb4ea1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 718d70a5aef745a8bd1ac21cb473cbba in term 1.
I20260926 20:59:51.418026 18609 raft_consensus.cc:2454] T 9e05207caf894f12a5e0a9f1f08363b4 P 187ea4ac3cf744a6af66159f04cfc837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 718d70a5aef745a8bd1ac21cb473cbba in term 1.
I20260926 20:59:51.418190 18810 leader_election.cc:302] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34d75d07a15645518bb3abb82fb4ea1e, 718d70a5aef745a8bd1ac21cb473cbba; no voters:
I20260926 20:59:51.418310 18945 raft_consensus.cc:2781] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:59:51.418470 18945 raft_consensus.cc:686] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [term 1 LEADER]: Becoming Leader. State: Replica: 718d70a5aef745a8bd1ac21cb473cbba, State: Running, Role: LEADER
I20260926 20:59:51.418576 18945 consensus_queue.cc:237] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d75d07a15645518bb3abb82fb4ea1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37203 } } peers { permanent_uuid: "187ea4ac3cf744a6af66159f04cfc837" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46745 } } peers { permanent_uuid: "718d70a5aef745a8bd1ac21cb473cbba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39031 } }
I20260926 20:59:51.419441 18464 catalog_manager.cc:5495] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba reported cstate change: term changed from 0 to 1, leader changed from <none> to 718d70a5aef745a8bd1ac21cb473cbba (127.0.98.3). New cstate: current_term: 1 leader_uuid: "718d70a5aef745a8bd1ac21cb473cbba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d75d07a15645518bb3abb82fb4ea1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "187ea4ac3cf744a6af66159f04cfc837" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "718d70a5aef745a8bd1ac21cb473cbba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39031 } health_report { overall_health: HEALTHY } } }
I20260926 20:59:51.442147 18937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.439710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.0.98.3:39031)
0926 20:59:51.439788 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:51.439790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:51.439799 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:59:51.439835 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:51.439835 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:51.439853 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:51.439924 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:59:51.440099 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.440648 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.441467 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.441655 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.441711 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.441855 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:59:51.441945 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:59:51.442017 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:51.442055 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 20:59:51.442062 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:59:51.444240 18609 raft_consensus.cc:1261] T 9e05207caf894f12a5e0a9f1f08363b4 P 187ea4ac3cf744a6af66159f04cfc837 [term 1 FOLLOWER]: Refusing update from remote peer 718d70a5aef745a8bd1ac21cb473cbba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:59:51.444240 18743 raft_consensus.cc:1261] T 9e05207caf894f12a5e0a9f1f08363b4 P 34d75d07a15645518bb3abb82fb4ea1e [term 1 FOLLOWER]: Refusing update from remote peer 718d70a5aef745a8bd1ac21cb473cbba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:59:51.444765 18950 consensus_queue.cc:1035] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "187ea4ac3cf744a6af66159f04cfc837" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:59:51.444926 18945 consensus_queue.cc:1035] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "34d75d07a15645518bb3abb82fb4ea1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:59:51.447425 18953 mvcc.cc:205] Tried to move back new op lower bound from 7333709379352252416 to 7333709379251277824. Current Snapshot: MvccSnapshot[applied={T|T < 7333709379352252416}]
I20260926 20:59:51.447852 18955 mvcc.cc:205] Tried to move back new op lower bound from 7333709379352252416 to 7333709379251277824. Current Snapshot: MvccSnapshot[applied={T|T < 7333709379352252416 or (T in {7333709379352252416})}]
I20260926 20:59:51.448364 18954 mvcc.cc:205] Tried to move back new op lower bound from 7333709379352252416 to 7333709379251277824. Current Snapshot: MvccSnapshot[applied={T|T < 7333709379352252416}]
I20260926 20:59:51.450007 18464 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:59:51.450119 18464 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:59:51.450847 18464 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64187354f7e8488da5941336382dcf68: Sending DeleteTablet for 3 replicas of tablet 9e05207caf894f12a5e0a9f1f08363b4
I20260926 20:59:51.451205 18857 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e05207caf894f12a5e0a9f1f08363b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55118
I20260926 20:59:51.451211 18589 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e05207caf894f12a5e0a9f1f08363b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60806
I20260926 20:59:51.451469 18723 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e05207caf894f12a5e0a9f1f08363b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56666
I20260926 20:59:51.451684 18962 tablet_replica.cc:330] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba: stopping tablet replica
I20260926 20:59:51.451774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 18526
I20260926 20:59:51.452486 18962 raft_consensus.cc:2229] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:59:51.452597 18962 raft_consensus.cc:2258] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:59:51.453078 18962 ts_tablet_manager.cc:1905] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:59:51.455178 18962 ts_tablet_manager.cc:1918] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:59:51.455268 18962 log.cc:1196] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/9e05207caf894f12a5e0a9f1f08363b4
I20260926 20:59:51.455550 18962 ts_tablet_manager.cc:1939] T 9e05207caf894f12a5e0a9f1f08363b4 P 718d70a5aef745a8bd1ac21cb473cbba: Deleting consensus metadata
I20260926 20:59:51.456101 18449 catalog_manager.cc:4841] TS 718d70a5aef745a8bd1ac21cb473cbba (127.0.98.3:39031): tablet 9e05207caf894f12a5e0a9f1f08363b4 (table test-table [id=54eb6b1508e74cedb2dfabc1c01cc819]) successfully deleted
W20260926 20:59:51.458968 18448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46745 (error 108)
I20260926 20:59:51.459147 18964 tablet_replica.cc:330] T 9e05207caf894f12a5e0a9f1f08363b4 P 34d75d07a15645518bb3abb82fb4ea1e: stopping tablet replica
I20260926 20:59:51.459350 18964 raft_consensus.cc:2229] T 9e05207caf894f12a5e0a9f1f08363b4 P 34d75d07a15645518bb3abb82fb4ea1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:59:51.459507 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.459373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46745 (local address 127.0.0.1:60822)
0926 20:59:51.459430 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:51.459466 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 20:59:51.459542 18964 raft_consensus.cc:2258] T 9e05207caf894f12a5e0a9f1f08363b4 P 34d75d07a15645518bb3abb82fb4ea1e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 20:59:51.459618 18448 catalog_manager.cc:4584] TS 187ea4ac3cf744a6af66159f04cfc837 (127.0.98.1:46745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e05207caf894f12a5e0a9f1f08363b4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46745: connect: Connection refused (error 111)
I20260926 20:59:51.459911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 18660
I20260926 20:59:51.468052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 18794
I20260926 20:59:51.468207 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.468063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37203 (local address 127.0.0.1:56678)
0926 20:59:51.468126 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:51.468159 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 20:59:51.473182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 18432
2026-09-26T20:59:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2082 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T20:59:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:59:51Z Disabled control of system clock
I20260926 20:59:51.501852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42733
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:59:51.585116 18969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:51.585358 18969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:51.585381 18969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:51.586583 18969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:59:51.586629 18969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:51.586642 18969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:51.586655 18969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:59:51.586668 18969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:59:51.588021 18969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42733
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:51.588238 18969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:51.588526 18969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:51.593618 18977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:51.593624 18974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:51.593659 18969 server_base.cc:1034] running on GCE node
W20260926 20:59:51.593624 18975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:51.594187 18969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:51.594512 18969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:51.595713 18969 hybrid_clock.cc:648] HybridClock initialized: now 1790456391595700 us; error 44 us; skew 500 ppm
I20260926 20:59:51.597146 18969 webserver.cc:460] Webserver started at http://127.0.98.62:39019/ using document root <none> and password file <none>
I20260926 20:59:51.597432 18969 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:51.597512 18969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:51.597678 18969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:51.598809 18969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "f44027b08bb84041b78372389b8e667f"
format_stamp: "Formatted at 2026-09-26 20:59:51 on dist-test-slave-fl7g"
server_key: "2ca9851a0ab5451696237357797b00e9"
server_key_iv: "2649fd70ec2bdc192738390d11349a63"
server_key_version: "encryptionkey@0"
I20260926 20:59:51.599237 18969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "f44027b08bb84041b78372389b8e667f"
format_stamp: "Formatted at 2026-09-26 20:59:51 on dist-test-slave-fl7g"
server_key: "2ca9851a0ab5451696237357797b00e9"
server_key_iv: "2649fd70ec2bdc192738390d11349a63"
server_key_version: "encryptionkey@0"
I20260926 20:59:51.600771 18969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:59:51.601593 18983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:51.601785 18969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:51.601893 18969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "f44027b08bb84041b78372389b8e667f"
format_stamp: "Formatted at 2026-09-26 20:59:51 on dist-test-slave-fl7g"
server_key: "2ca9851a0ab5451696237357797b00e9"
server_key_iv: "2649fd70ec2bdc192738390d11349a63"
server_key_version: "encryptionkey@0"
I20260926 20:59:51.601969 18969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:51.614027 18969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:51.614398 18969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:51.614573 18969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:51.618873 18969 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42733
I20260926 20:59:51.618918 19035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42733 every 8 connection(s)
I20260926 20:59:51.619315 18969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:59:51.620376 19036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:51.622912 19036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f: Bootstrap starting.
I20260926 20:59:51.623553 19036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:51.623888 19036 log.cc:824] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:51.624701 19036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f: No bootstrap required, opened a new log
I20260926 20:59:51.626439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 18969
I20260926 20:59:51.626550 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:59:51.626623 19036 raft_consensus.cc:348] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44027b08bb84041b78372389b8e667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } }
I20260926 20:59:51.626784 392 external_mini_cluster.cc:1428] Setting key 0683af30209f6f3cbc09597d53512ac3
I20260926 20:59:51.626829 19036 raft_consensus.cc:374] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:51.626871 19036 raft_consensus.cc:729] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f44027b08bb84041b78372389b8e667f, State: Initialized, Role: FOLLOWER
I20260926 20:59:51.627044 19036 consensus_queue.cc:260] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44027b08bb84041b78372389b8e667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } }
I20260926 20:59:51.627127 19036 raft_consensus.cc:388] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:59:51.627153 19036 raft_consensus.cc:482] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:59:51.627228 19036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:51.627925 19036 raft_consensus.cc:504] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44027b08bb84041b78372389b8e667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } }
I20260926 20:59:51.628059 19036 leader_election.cc:302] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f44027b08bb84041b78372389b8e667f; no voters:
I20260926 20:59:51.628294 19036 leader_election.cc:288] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:59:51.628326 19041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:59:51.628476 19041 raft_consensus.cc:686] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f [term 1 LEADER]: Becoming Leader. State: Replica: f44027b08bb84041b78372389b8e667f, State: Running, Role: LEADER
I20260926 20:59:51.628574 19041 consensus_queue.cc:237] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44027b08bb84041b78372389b8e667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } }
I20260926 20:59:51.628630 19036 sys_catalog.cc:564] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:59:51.628984 19041 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f44027b08bb84041b78372389b8e667f. Latest consensus state: current_term: 1 leader_uuid: "f44027b08bb84041b78372389b8e667f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44027b08bb84041b78372389b8e667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } }
I20260926 20:59:51.629073 19041 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:51.629907 19047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:59:51.630367 19042 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f44027b08bb84041b78372389b8e667f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44027b08bb84041b78372389b8e667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } }
I20260926 20:59:51.630466 19042 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:51.630524 19047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:59:51.632346 19047 catalog_manager.cc:1347] Generated new cluster ID: d629173cc87b4ca59ede674b5553ba45
I20260926 20:59:51.632438 19047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:59:51.634123 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.627338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.0.98.62:42733)
0926 20:59:51.627606 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:51.627622 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:51.627655 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:59:51.627766 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:51.627772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:51.627837 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:51.627977 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:51.628227 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.630649 (+ 2422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.631735 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.632356 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.632541 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.633365 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:51.633369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:51.633369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:51.633586 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:51.633630 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:51.633646 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:51.633716 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:51.633806 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 20:59:51.633886 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":113,"threads_started":1}
I20260926 20:59:51.642128 19047 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:59:51.642910 19047 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:59:51.650971 19047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f: Generated new TSK 0
I20260926 20:59:51.651209 19047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:59:51.656026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42733
--builtin_ntp_servers=127.0.98.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:59:51.738148 19060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:51.738420 19060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:51.738461 19060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:51.739666 19060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:51.739734 19060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:51.739784 19060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:59:51.741304 19060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:51.741583 19060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:51.741902 19060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:51.747925 19066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:51.747936 19065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:51.747942 19068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:51.748215 19060 server_base.cc:1034] running on GCE node
I20260926 20:59:51.748538 19060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:51.748795 19060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:51.749963 19060 hybrid_clock.cc:648] HybridClock initialized: now 1790456391749938 us; error 43 us; skew 500 ppm
I20260926 20:59:51.751225 19060 webserver.cc:460] Webserver started at http://127.0.98.1:39765/ using document root <none> and password file <none>
I20260926 20:59:51.751482 19060 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:51.751554 19060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:51.751720 19060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:51.752892 19060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "428da94bb2664ca884b633c731ca749d"
format_stamp: "Formatted at 2026-09-26 20:59:51 on dist-test-slave-fl7g"
server_key: "d655f3505d5f03927d9346153d99a9a6"
server_key_iv: "49063a5a8146fb1479d5676609ab808d"
server_key_version: "encryptionkey@0"
I20260926 20:59:51.753254 19060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "428da94bb2664ca884b633c731ca749d"
format_stamp: "Formatted at 2026-09-26 20:59:51 on dist-test-slave-fl7g"
server_key: "d655f3505d5f03927d9346153d99a9a6"
server_key_iv: "49063a5a8146fb1479d5676609ab808d"
server_key_version: "encryptionkey@0"
I20260926 20:59:51.754606 19060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:59:51.755389 19074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:51.755559 19060 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:59:51.755656 19060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "428da94bb2664ca884b633c731ca749d"
format_stamp: "Formatted at 2026-09-26 20:59:51 on dist-test-slave-fl7g"
server_key: "d655f3505d5f03927d9346153d99a9a6"
server_key_iv: "49063a5a8146fb1479d5676609ab808d"
server_key_version: "encryptionkey@0"
I20260926 20:59:51.755738 19060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:51.781011 19060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:51.781368 19060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:51.781539 19060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:51.781805 19060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:51.782151 19060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:51.782207 19060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:51.782253 19060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:51.782277 19060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:51.788066 19060 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44387
I20260926 20:59:51.788105 19187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44387 every 8 connection(s)
I20260926 20:59:51.788514 19060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:59:51.791285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 19060
I20260926 20:59:51.791393 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:59:51.791579 392 external_mini_cluster.cc:1428] Setting key fc7fd97a777529b857b96c3f17b3838c
I20260926 20:59:51.792584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42733
--builtin_ntp_servers=127.0.98.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:59:51.794893 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.789268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36697 (local address 127.0.98.62:42733)
0926 20:59:51.789349 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:51.789351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:51.789591 (+ 240us) server_negotiation.cc:407] Connection header received
0926 20:59:51.790166 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:51.790167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:51.790181 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:51.790231 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:51.790933 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.791743 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.793174 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.793460 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.793529 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.794557 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:51.794559 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:51.794559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:51.794564 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:51.794588 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:51.794592 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:51.794652 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:51.794772 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 20:59:51.794812 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:59:51.795117 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.789251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42733 (local address 127.0.98.1:36697)
0926 20:59:51.789464 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:51.789497 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:51.790053 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:51.790300 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:51.790304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:51.790315 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:51.790832 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:51.790837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.791868 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.791869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:51.793088 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:51.793091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.793551 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.793553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:51.793726 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.794395 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:59:51.794395 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:51.794426 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:59:51.794450 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:51.794652 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:51.794659 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:51.794663 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:59:51.794760 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 20:59:51.794886 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260926 20:59:51.795737 19188 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42733
I20260926 20:59:51.795836 19188 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:51.796043 19188 heartbeater.cc:511] Master 127.0.98.62:42733 requested a full tablet report, sending...
I20260926 20:59:51.796636 19000 ts_manager.cc:194] Registered new tserver with Master: 428da94bb2664ca884b633c731ca749d (127.0.98.1:44387)
I20260926 20:59:51.797216 19000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36697
W20260926 20:59:51.875129 19191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:51.875402 19191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:51.875442 19191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:51.876674 19191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:51.876754 19191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:51.876796 19191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:59:51.878187 19191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:51.878588 19191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:51.878953 19191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:51.884215 19199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:51.884450 19191 server_base.cc:1034] running on GCE node
W20260926 20:59:51.884210 19196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:51.884210 19197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:51.884737 19191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:51.885010 19191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:51.886252 19191 hybrid_clock.cc:648] HybridClock initialized: now 1790456391886235 us; error 47 us; skew 500 ppm
I20260926 20:59:51.887753 19191 webserver.cc:460] Webserver started at http://127.0.98.2:41579/ using document root <none> and password file <none>
I20260926 20:59:51.888041 19191 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:51.888088 19191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:51.888195 19191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:51.889380 19191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "21e2d87c8e454fe0be196b1012f14104"
format_stamp: "Formatted at 2026-09-26 20:59:51 on dist-test-slave-fl7g"
server_key: "294be3e6296c65c76c207ba93355f81f"
server_key_iv: "b3b955d89350fbb2415bac3ce52ce581"
server_key_version: "encryptionkey@0"
I20260926 20:59:51.889709 19191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "21e2d87c8e454fe0be196b1012f14104"
format_stamp: "Formatted at 2026-09-26 20:59:51 on dist-test-slave-fl7g"
server_key: "294be3e6296c65c76c207ba93355f81f"
server_key_iv: "b3b955d89350fbb2415bac3ce52ce581"
server_key_version: "encryptionkey@0"
I20260926 20:59:51.890995 19191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:59:51.891695 19205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:51.891861 19191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:59:51.891954 19191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "21e2d87c8e454fe0be196b1012f14104"
format_stamp: "Formatted at 2026-09-26 20:59:51 on dist-test-slave-fl7g"
server_key: "294be3e6296c65c76c207ba93355f81f"
server_key_iv: "b3b955d89350fbb2415bac3ce52ce581"
server_key_version: "encryptionkey@0"
I20260926 20:59:51.892012 19191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:51.910321 19191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:51.910597 19191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:51.910701 19191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:51.910895 19191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:51.911293 19191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:51.911342 19191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:51.911391 19191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:51.911418 19191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:51.917111 19191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37091
I20260926 20:59:51.917204 19318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37091 every 8 connection(s)
I20260926 20:59:51.917536 19191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:59:51.918469 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 19191
I20260926 20:59:51.918581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:59:51.918810 392 external_mini_cluster.cc:1428] Setting key 0361c9cc03464fed460a5183197fd235
I20260926 20:59:51.919778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42733
--builtin_ntp_servers=127.0.98.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:59:51.923399 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.918387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55089 (local address 127.0.98.62:42733)
0926 20:59:51.919017 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:51.919024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:51.919036 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:59:51.919456 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:51.919457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:51.919470 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:51.919522 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:51.920338 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.920899 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.921908 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.922112 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.922198 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.923082 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:51.923083 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:51.923083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:51.923088 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:51.923103 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:51.923108 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:51.923181 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:51.923279 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 20:59:51.923318 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594}
I20260926 20:59:51.923610 19321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:51.918388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42733 (local address 127.0.98.2:55089)
0926 20:59:51.918636 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:51.918680 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:51.919345 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:51.919611 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:51.919620 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:51.919628 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:51.920240 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:51.920246 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.920997 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.920998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:51.921813 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:51.921815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:51.922208 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:51.922209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:51.922348 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:51.922922 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:59:51.922923 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:51.922953 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:59:51.922978 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:51.923185 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:51.923191 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:51.923194 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:59:51.923281 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 20:59:51.923383 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":99,"threads_started":1}
I20260926 20:59:51.924101 19319 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42733
I20260926 20:59:51.924197 19319 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:51.924448 19319 heartbeater.cc:511] Master 127.0.98.62:42733 requested a full tablet report, sending...
I20260926 20:59:51.924865 19000 ts_manager.cc:194] Registered new tserver with Master: 21e2d87c8e454fe0be196b1012f14104 (127.0.98.2:37091)
I20260926 20:59:51.925426 19000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:55089
W20260926 20:59:52.001576 19322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:52.001806 19322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:52.001829 19322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:52.002946 19322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:52.002979 19322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:52.003005 19322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:59:52.004426 19322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:52.004688 19322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:52.004997 19322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:52.010334 19327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:52.010334 19330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:52.010375 19328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:52.010766 19322 server_base.cc:1034] running on GCE node
I20260926 20:59:52.011025 19322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:52.011308 19322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:52.012485 19322 hybrid_clock.cc:648] HybridClock initialized: now 1790456392012476 us; error 41 us; skew 500 ppm
I20260926 20:59:52.013850 19322 webserver.cc:460] Webserver started at http://127.0.98.3:40079/ using document root <none> and password file <none>
I20260926 20:59:52.014115 19322 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:52.014189 19322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:52.014357 19322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:52.015486 19322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "1188cba6756e4b68b070c5c1f94c0d50"
format_stamp: "Formatted at 2026-09-26 20:59:52 on dist-test-slave-fl7g"
server_key: "27e3b98f177877467b5d679a50d43713"
server_key_iv: "50ff406201cf5561fcbf03519f243520"
server_key_version: "encryptionkey@0"
I20260926 20:59:52.015864 19322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1188cba6756e4b68b070c5c1f94c0d50"
format_stamp: "Formatted at 2026-09-26 20:59:52 on dist-test-slave-fl7g"
server_key: "27e3b98f177877467b5d679a50d43713"
server_key_iv: "50ff406201cf5561fcbf03519f243520"
server_key_version: "encryptionkey@0"
I20260926 20:59:52.017402 19322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:59:52.018242 19336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:52.018428 19322 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:52.018532 19322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "1188cba6756e4b68b070c5c1f94c0d50"
format_stamp: "Formatted at 2026-09-26 20:59:52 on dist-test-slave-fl7g"
server_key: "27e3b98f177877467b5d679a50d43713"
server_key_iv: "50ff406201cf5561fcbf03519f243520"
server_key_version: "encryptionkey@0"
I20260926 20:59:52.018615 19322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:52.028685 19322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:52.029013 19322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:52.029172 19322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:52.029430 19322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:52.029832 19322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:52.029891 19322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:52.029927 19322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:52.029958 19322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:52.035928 19322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43657
I20260926 20:59:52.035979 19449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43657 every 8 connection(s)
I20260926 20:59:52.036398 19322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:59:52.042294 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:52.037315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43381 (local address 127.0.98.62:42733)
0926 20:59:52.037389 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:52.037391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:52.037694 (+ 303us) server_negotiation.cc:407] Connection header received
0926 20:59:52.038213 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:52.038214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:52.038226 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:52.038288 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:52.039087 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.039653 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.040738 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.040953 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.041019 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:52.042024 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:52.042026 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:52.042026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:52.042031 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:52.042047 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:52.042051 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:52.042101 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:52.042196 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 20:59:52.042225 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:59:52.042546 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:52.037319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42733 (local address 127.0.98.3:43381)
0926 20:59:52.037546 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:52.037583 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:52.038099 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:52.038378 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:52.038381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:52.038390 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:52.038946 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:52.038951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.039781 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.039782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:52.040628 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:52.040631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.041059 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.041060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:52.041188 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:52.041868 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:59:52.041869 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:52.041902 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:59:52.041926 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:52.042115 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:52.042119 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:52.042121 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:59:52.042176 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 20:59:52.042271 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":105,"threads_started":1}
I20260926 20:59:52.043135 19450 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42733
I20260926 20:59:52.043273 19450 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:52.043444 19450 heartbeater.cc:511] Master 127.0.98.62:42733 requested a full tablet report, sending...
I20260926 20:59:52.043815 19000 ts_manager.cc:194] Registered new tserver with Master: 1188cba6756e4b68b070c5c1f94c0d50 (127.0.98.3:43657)
I20260926 20:59:52.044201 19000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:43381
I20260926 20:59:52.044664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 19322
I20260926 20:59:52.044812 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:59:52.045027 392 external_mini_cluster.cc:1428] Setting key 0dc993a53d525d6c51774db07afe1d39
I20260926 20:59:52.045849 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:59:52.050088 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:52.047217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44750 (local address 127.0.98.62:42733)
0926 20:59:52.047281 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:52.047283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:52.047415 (+ 132us) server_negotiation.cc:407] Connection header received
0926 20:59:52.047452 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:52.047452 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:52.047463 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:52.047514 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:52.047684 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.048147 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.048887 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.049063 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.049115 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:52.049825 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:52.049826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:52.049826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:52.049830 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:52.049840 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:52.049844 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:52.049901 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:52.049958 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 20:59:52.049989 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:59:52.052098 19000 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:59:52.053773 19000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:59:52.060667 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:52.056047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37091 (local address 127.0.0.1:38570)
0926 20:59:52.056830 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:52.056840 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:52.056886 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:52.057089 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:52.057090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:52.057092 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:52.057126 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:52.057127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.058355 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.058357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:52.059428 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:52.059431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.060347 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.060348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:52.060452 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:52.060469 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 20:59:52.060538 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:59:52.060585 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":121,"threads_started":1}
I20260926 20:59:52.060797 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:52.056037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43657 (local address 127.0.0.1:55862)
0926 20:59:52.056219 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:52.056258 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:52.056330 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:52.056851 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:52.056852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:52.056854 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:52.056914 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:52.056916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.058330 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.058330 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:52.059397 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:52.059400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.060558 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.060559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:52.060618 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:52.060622 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:59:52.060679 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 20:59:52.060721 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":1,"thread_start_us":118,"threads_started":1}
I20260926 20:59:52.060810 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:52.056447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55862 (local address 127.0.98.3:43657)
0926 20:59:52.056624 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:52.056628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:52.056636 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:59:52.056673 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:52.056674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:52.056689 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:52.056991 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:52.057083 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.058232 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.059476 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.060427 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.060548 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:52.060643 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:52.060695 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 20:59:52.060730 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260926 20:59:52.060907 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:52.056296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38570 (local address 127.0.98.2:37091)
0926 20:59:52.056540 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:52.056545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:52.056953 (+ 408us) server_negotiation.cc:407] Connection header received
0926 20:59:52.056999 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:52.057001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:52.057015 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:52.057089 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:52.057188 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.058261 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.059522 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.060254 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.060342 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:52.060465 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:52.060788 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 20:59:52.060832 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":85,"threads_started":1}
I20260926 20:59:52.061663 19384 tablet_service.cc:1467] Processing CreateTablet for tablet 896922d8070241c485ea1539bb21f2d7 (DEFAULT_TABLE table=test-table [id=b95f0bfb307d470da25a8f74661f678a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:52.061849 19253 tablet_service.cc:1467] Processing CreateTablet for tablet 896922d8070241c485ea1539bb21f2d7 (DEFAULT_TABLE table=test-table [id=b95f0bfb307d470da25a8f74661f678a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:52.061986 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:52.056037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44387 (local address 127.0.0.1:40656)
0926 20:59:52.056219 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:52.056265 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:52.056332 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:52.056760 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:52.056764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:52.056767 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:52.056849 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:52.056851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.057850 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.057851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:52.060132 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:52.060135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.061768 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.061769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:52.061829 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:52.061832 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:59:52.061877 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 20:59:52.061915 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":4,"thread_start_us":64,"threads_started":1}
I20260926 20:59:52.062054 19384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896922d8070241c485ea1539bb21f2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:52.062156 19253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896922d8070241c485ea1539bb21f2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:52.062287 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:52.056233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.0.98.1:44387)
0926 20:59:52.056557 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:52.056561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:52.056573 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:59:52.056616 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:52.056618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:52.056636 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:52.056724 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:52.056939 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.057767 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.060217 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.061667 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.061765 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:52.061929 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:52.061955 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 20:59:52.061991 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":77,"threads_started":1}
I20260926 20:59:52.065078 19122 tablet_service.cc:1467] Processing CreateTablet for tablet 896922d8070241c485ea1539bb21f2d7 (DEFAULT_TABLE table=test-table [id=b95f0bfb307d470da25a8f74661f678a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:52.065438 19122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896922d8070241c485ea1539bb21f2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:52.065647 19465 tablet_bootstrap.cc:492] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50: Bootstrap starting.
I20260926 20:59:52.066329 19465 tablet_bootstrap.cc:654] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:52.066655 19465 log.cc:824] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:52.066943 19464 tablet_bootstrap.cc:492] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104: Bootstrap starting.
I20260926 20:59:52.067495 19465 tablet_bootstrap.cc:492] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50: No bootstrap required, opened a new log
I20260926 20:59:52.067641 19465 ts_tablet_manager.cc:1397] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:59:52.067720 19464 tablet_bootstrap.cc:654] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:52.068049 19464 log.cc:824] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:52.068892 19464 tablet_bootstrap.cc:492] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104: No bootstrap required, opened a new log
I20260926 20:59:52.068962 19464 ts_tablet_manager.cc:1397] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 20:59:52.069144 19468 tablet_bootstrap.cc:492] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d: Bootstrap starting.
I20260926 20:59:52.069846 19468 tablet_bootstrap.cc:654] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:52.070223 19468 log.cc:824] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:52.070345 19465 raft_consensus.cc:348] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1188cba6756e4b68b070c5c1f94c0d50" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43657 } } peers { permanent_uuid: "428da94bb2664ca884b633c731ca749d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44387 } } peers { permanent_uuid: "21e2d87c8e454fe0be196b1012f14104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } }
I20260926 20:59:52.070519 19465 raft_consensus.cc:374] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:52.070564 19465 raft_consensus.cc:729] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1188cba6756e4b68b070c5c1f94c0d50, State: Initialized, Role: FOLLOWER
I20260926 20:59:52.070676 19465 consensus_queue.cc:260] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1188cba6756e4b68b070c5c1f94c0d50" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43657 } } peers { permanent_uuid: "428da94bb2664ca884b633c731ca749d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44387 } } peers { permanent_uuid: "21e2d87c8e454fe0be196b1012f14104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } }
I20260926 20:59:52.070978 19465 ts_tablet_manager.cc:1428] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 20:59:52.071034 19450 heartbeater.cc:503] Master 127.0.98.62:42733 was elected leader, sending a full tablet report...
I20260926 20:59:52.071054 19464 raft_consensus.cc:348] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1188cba6756e4b68b070c5c1f94c0d50" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43657 } } peers { permanent_uuid: "428da94bb2664ca884b633c731ca749d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44387 } } peers { permanent_uuid: "21e2d87c8e454fe0be196b1012f14104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } }
I20260926 20:59:52.071211 19464 raft_consensus.cc:374] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:52.071247 19464 raft_consensus.cc:729] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21e2d87c8e454fe0be196b1012f14104, State: Initialized, Role: FOLLOWER
I20260926 20:59:52.071370 19464 consensus_queue.cc:260] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1188cba6756e4b68b070c5c1f94c0d50" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43657 } } peers { permanent_uuid: "428da94bb2664ca884b633c731ca749d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44387 } } peers { permanent_uuid: "21e2d87c8e454fe0be196b1012f14104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } }
I20260926 20:59:52.071558 19468 tablet_bootstrap.cc:492] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d: No bootstrap required, opened a new log
I20260926 20:59:52.071628 19468 ts_tablet_manager.cc:1397] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 20:59:52.071684 19319 heartbeater.cc:503] Master 127.0.98.62:42733 was elected leader, sending a full tablet report...
I20260926 20:59:52.071640 19464 ts_tablet_manager.cc:1428] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:59:52.073760 19468 raft_consensus.cc:348] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1188cba6756e4b68b070c5c1f94c0d50" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43657 } } peers { permanent_uuid: "428da94bb2664ca884b633c731ca749d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44387 } } peers { permanent_uuid: "21e2d87c8e454fe0be196b1012f14104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } }
I20260926 20:59:52.073902 19468 raft_consensus.cc:374] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:52.073926 19468 raft_consensus.cc:729] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428da94bb2664ca884b633c731ca749d, State: Initialized, Role: FOLLOWER
I20260926 20:59:52.074043 19468 consensus_queue.cc:260] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1188cba6756e4b68b070c5c1f94c0d50" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43657 } } peers { permanent_uuid: "428da94bb2664ca884b633c731ca749d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44387 } } peers { permanent_uuid: "21e2d87c8e454fe0be196b1012f14104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } }
I20260926 20:59:52.074256 19468 ts_tablet_manager.cc:1428] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:59:52.074327 19188 heartbeater.cc:503] Master 127.0.98.62:42733 was elected leader, sending a full tablet report...
W20260926 20:59:52.168120 19320 tablet.cc:2369] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:59:52.243199 19472 raft_consensus.cc:482] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:59:52.243332 19472 raft_consensus.cc:504] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1188cba6756e4b68b070c5c1f94c0d50" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43657 } } peers { permanent_uuid: "428da94bb2664ca884b633c731ca749d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44387 } } peers { permanent_uuid: "21e2d87c8e454fe0be196b1012f14104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } }
I20260926 20:59:52.243649 19472 leader_election.cc:288] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1188cba6756e4b68b070c5c1f94c0d50 (127.0.98.3:43657), 21e2d87c8e454fe0be196b1012f14104 (127.0.98.2:37091)
I20260926 20:59:52.247501 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:52.243884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43657 (local address 127.0.98.1:38419)
0926 20:59:52.243934 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:52.243949 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:52.243999 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:52.244208 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:52.244209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:52.244211 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:52.244289 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:52.244290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.245232 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.245233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:52.246478 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:52.246481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.247207 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.247208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:52.247302 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:52.247306 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:59:52.247362 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:59:52.247412 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 20:59:52.247545 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:52.244011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38419 (local address 127.0.98.3:43657)
0926 20:59:52.244057 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:52.244059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:52.244069 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:59:52.244101 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:52.244102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:52.244118 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:52.244203 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:52.244373 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.245152 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.246559 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.247123 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.247198 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:52.247209 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:52.247409 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 20:59:52.247479 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:59:52.247586 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:52.244023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48289 (local address 127.0.98.2:37091)
0926 20:59:52.244065 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:52.244067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:52.244382 (+ 315us) server_negotiation.cc:407] Connection header received
0926 20:59:52.244413 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:52.244415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:52.244447 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:52.244490 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:52.244653 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.245313 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.246635 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.247203 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.247266 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:52.247274 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:52.247476 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 20:59:52.247519 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:59:52.247607 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:52.243927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37091 (local address 127.0.98.1:48289)
0926 20:59:52.244104 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:52.244121 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:52.244171 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:52.244516 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:52.244517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:52.244519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:52.244583 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:52.244584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.245397 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.245398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:52.246556 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:52.246558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.247286 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.247287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:52.247360 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:52.247363 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:59:52.247444 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 20:59:52.247474 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":122,"threads_started":1}
I20260926 20:59:52.247951 19404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896922d8070241c485ea1539bb21f2d7" candidate_uuid: "428da94bb2664ca884b633c731ca749d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1188cba6756e4b68b070c5c1f94c0d50" is_pre_election: true
I20260926 20:59:52.248005 19273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896922d8070241c485ea1539bb21f2d7" candidate_uuid: "428da94bb2664ca884b633c731ca749d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21e2d87c8e454fe0be196b1012f14104" is_pre_election: true
I20260926 20:59:52.248138 19404 raft_consensus.cc:2454] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428da94bb2664ca884b633c731ca749d in term 0.
I20260926 20:59:52.248168 19273 raft_consensus.cc:2454] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428da94bb2664ca884b633c731ca749d in term 0.
I20260926 20:59:52.248397 19076 leader_election.cc:302] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1188cba6756e4b68b070c5c1f94c0d50, 428da94bb2664ca884b633c731ca749d; no voters:
I20260926 20:59:52.248558 19472 raft_consensus.cc:2781] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:59:52.248616 19472 raft_consensus.cc:482] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:59:52.248647 19472 raft_consensus.cc:3037] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:52.249329 19472 raft_consensus.cc:504] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1188cba6756e4b68b070c5c1f94c0d50" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43657 } } peers { permanent_uuid: "428da94bb2664ca884b633c731ca749d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44387 } } peers { permanent_uuid: "21e2d87c8e454fe0be196b1012f14104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } }
I20260926 20:59:52.249476 19472 leader_election.cc:288] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [CANDIDATE]: Term 1 election: Requested vote from peers 1188cba6756e4b68b070c5c1f94c0d50 (127.0.98.3:43657), 21e2d87c8e454fe0be196b1012f14104 (127.0.98.2:37091)
I20260926 20:59:52.249634 19273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896922d8070241c485ea1539bb21f2d7" candidate_uuid: "428da94bb2664ca884b633c731ca749d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21e2d87c8e454fe0be196b1012f14104"
I20260926 20:59:52.249640 19404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896922d8070241c485ea1539bb21f2d7" candidate_uuid: "428da94bb2664ca884b633c731ca749d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1188cba6756e4b68b070c5c1f94c0d50"
I20260926 20:59:52.249732 19404 raft_consensus.cc:3037] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:52.249711 19273 raft_consensus.cc:3037] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:52.250468 19404 raft_consensus.cc:2454] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428da94bb2664ca884b633c731ca749d in term 1.
I20260926 20:59:52.250481 19273 raft_consensus.cc:2454] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428da94bb2664ca884b633c731ca749d in term 1.
I20260926 20:59:52.250649 19076 leader_election.cc:302] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1188cba6756e4b68b070c5c1f94c0d50, 428da94bb2664ca884b633c731ca749d; no voters:
I20260926 20:59:52.250765 19472 raft_consensus.cc:2781] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:59:52.250906 19472 raft_consensus.cc:686] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [term 1 LEADER]: Becoming Leader. State: Replica: 428da94bb2664ca884b633c731ca749d, State: Running, Role: LEADER
I20260926 20:59:52.251012 19472 consensus_queue.cc:237] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1188cba6756e4b68b070c5c1f94c0d50" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43657 } } peers { permanent_uuid: "428da94bb2664ca884b633c731ca749d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44387 } } peers { permanent_uuid: "21e2d87c8e454fe0be196b1012f14104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } }
I20260926 20:59:52.251901 19000 catalog_manager.cc:5495] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d reported cstate change: term changed from 0 to 1, leader changed from <none> to 428da94bb2664ca884b633c731ca749d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "428da94bb2664ca884b633c731ca749d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1188cba6756e4b68b070c5c1f94c0d50" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428da94bb2664ca884b633c731ca749d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21e2d87c8e454fe0be196b1012f14104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 } health_report { overall_health: UNKNOWN } } }
W20260926 20:59:52.287081 19451 tablet.cc:2369] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:59:52.289247 19189 tablet.cc:2369] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:59:52.291437 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:52.288962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40672 (local address 127.0.98.1:44387)
0926 20:59:52.289019 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:52.289021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:52.289030 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:59:52.289086 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:52.289087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:52.289108 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:52.289216 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:59:52.289437 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.289924 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.290638 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:52.290820 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:52.290867 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:52.291036 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:59:52.291126 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:59:52.291201 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:52.291247 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 20:59:52.291256 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:59:52.293318 19273 raft_consensus.cc:1261] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104 [term 1 FOLLOWER]: Refusing update from remote peer 428da94bb2664ca884b633c731ca749d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:59:52.293318 19404 raft_consensus.cc:1261] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50 [term 1 FOLLOWER]: Refusing update from remote peer 428da94bb2664ca884b633c731ca749d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:59:52.293673 19472 consensus_queue.cc:1035] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1188cba6756e4b68b070c5c1f94c0d50" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:59:52.293846 19474 consensus_queue.cc:1035] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [LEADER]: Connected to new peer: Peer: permanent_uuid: "21e2d87c8e454fe0be196b1012f14104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:59:52.296707 19479 mvcc.cc:205] Tried to move back new op lower bound from 7333709382830497792 to 7333709382660849664. Current Snapshot: MvccSnapshot[applied={T|T < 7333709382830497792}]
I20260926 20:59:52.299161 18999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:59:52.299316 18999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:59:52.300266 18999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f44027b08bb84041b78372389b8e667f: Sending DeleteTablet for 3 replicas of tablet 896922d8070241c485ea1539bb21f2d7
I20260926 20:59:52.300631 19122 tablet_service.cc:1514] Processing DeleteTablet for tablet 896922d8070241c485ea1539bb21f2d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:52 UTC) from {username='slave'} at 127.0.0.1:40656
I20260926 20:59:52.300683 19253 tablet_service.cc:1514] Processing DeleteTablet for tablet 896922d8070241c485ea1539bb21f2d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:52 UTC) from {username='slave'} at 127.0.0.1:38570
I20260926 20:59:52.300904 19487 tablet_replica.cc:330] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d: stopping tablet replica
I20260926 20:59:52.300918 19384 tablet_service.cc:1514] Processing DeleteTablet for tablet 896922d8070241c485ea1539bb21f2d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:52 UTC) from {username='slave'} at 127.0.0.1:55862
I20260926 20:59:52.301018 19487 raft_consensus.cc:2229] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:59:52.301059 19488 tablet_replica.cc:330] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104: stopping tablet replica
I20260926 20:59:52.301167 19487 raft_consensus.cc:2258] T 896922d8070241c485ea1539bb21f2d7 P 428da94bb2664ca884b633c731ca749d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:59:52.301280 19488 raft_consensus.cc:2229] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:59:52.301498 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 19060
I20260926 20:59:52.302312 19488 raft_consensus.cc:2258] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:59:52.302922 19488 ts_tablet_manager.cc:1905] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:59:52.302930 19489 tablet_replica.cc:330] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50: stopping tablet replica
I20260926 20:59:52.303047 19489 raft_consensus.cc:2229] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:59:52.303159 19489 raft_consensus.cc:2258] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:59:52.303550 19489 ts_tablet_manager.cc:1905] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:59:52.305186 19489 ts_tablet_manager.cc:1918] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:59:52.305279 19489 log.cc:1196] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/896922d8070241c485ea1539bb21f2d7
I20260926 20:59:52.305328 19488 ts_tablet_manager.cc:1918] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:59:52.305401 19488 log.cc:1196] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/896922d8070241c485ea1539bb21f2d7
I20260926 20:59:52.305562 19489 ts_tablet_manager.cc:1939] T 896922d8070241c485ea1539bb21f2d7 P 1188cba6756e4b68b070c5c1f94c0d50: Deleting consensus metadata
I20260926 20:59:52.305651 19488 ts_tablet_manager.cc:1939] T 896922d8070241c485ea1539bb21f2d7 P 21e2d87c8e454fe0be196b1012f14104: Deleting consensus metadata
I20260926 20:59:52.306042 18985 catalog_manager.cc:4841] TS 1188cba6756e4b68b070c5c1f94c0d50 (127.0.98.3:43657): tablet 896922d8070241c485ea1539bb21f2d7 (table test-table [id=b95f0bfb307d470da25a8f74661f678a]) successfully deleted
I20260926 20:59:52.306226 18986 catalog_manager.cc:4841] TS 21e2d87c8e454fe0be196b1012f14104 (127.0.98.2:37091): tablet 896922d8070241c485ea1539bb21f2d7 (table test-table [id=b95f0bfb307d470da25a8f74661f678a]) successfully deleted
W20260926 20:59:52.307216 18984 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44387 (error 108)
I20260926 20:59:52.307833 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:52.307580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44387 (local address 127.0.0.1:40674)
0926 20:59:52.307739 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:52.307796 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 20:59:52.307837 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 19191
W20260926 20:59:52.308470 18984 catalog_manager.cc:4584] TS 428da94bb2664ca884b633c731ca749d (127.0.98.1:44387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 896922d8070241c485ea1539bb21f2d7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44387: connect: Connection refused (error 111)
I20260926 20:59:52.313251 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 19322
I20260926 20:59:52.318470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 18969
2026-09-26T20:59:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (844 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:59:52 dist-test-slave-fl7g krb5kdc[19493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:59:52 dist-test-slave-fl7g krb5kdc[19493](info): set up 2 sockets
Sep 26 20:59:52 dist-test-slave-fl7g krb5kdc[19493](info): commencing operation
krb5kdc: starting...
W20260926 20:59:53.364274 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:59:53 dist-test-slave-fl7g krb5kdc[19493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:59:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:59:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:59:53.488777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43007
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:53.571259 19506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:53.571540 19506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:53.571580 19506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:53.572921 19506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:59:53.573000 19506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:53.573024 19506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:53.573046 19506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:59:53.573076 19506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:59:53.574672 19506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:53.574968 19506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:53.575215 19506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:53.580299 19514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:53.580317 19512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:53.580315 19511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:53.580356 19506 server_base.cc:1034] running on GCE node
I20260926 20:59:53.580829 19506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:53.581182 19506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:53.582388 19506 hybrid_clock.cc:648] HybridClock initialized: now 1790456393582370 us; error 47 us; skew 500 ppm
Sep 26 20:59:53 dist-test-slave-fl7g krb5kdc[19493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:53.585151 19506 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:59:53.585683 19506 webserver.cc:460] Webserver started at http://127.0.98.62:35417/ using document root <none> and password file <none>
I20260926 20:59:53.585944 19506 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:53.586016 19506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:53.586165 19506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:53.587299 19506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "6a7cdbdbfaee4875b9fcb634d6b94e26"
format_stamp: "Formatted at 2026-09-26 20:59:53 on dist-test-slave-fl7g"
server_key: "2ea5a8f9b8927684dab77d72cc14d880"
server_key_iv: "5ece7875b5fd9407cdb6ee4e2dc534d8"
server_key_version: "encryptionkey@0"
I20260926 20:59:53.587685 19506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "6a7cdbdbfaee4875b9fcb634d6b94e26"
format_stamp: "Formatted at 2026-09-26 20:59:53 on dist-test-slave-fl7g"
server_key: "2ea5a8f9b8927684dab77d72cc14d880"
server_key_iv: "5ece7875b5fd9407cdb6ee4e2dc534d8"
server_key_version: "encryptionkey@0"
I20260926 20:59:53.589188 19506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:59:53.589973 19521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:53.590152 19506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:53.590246 19506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "6a7cdbdbfaee4875b9fcb634d6b94e26"
format_stamp: "Formatted at 2026-09-26 20:59:53 on dist-test-slave-fl7g"
server_key: "2ea5a8f9b8927684dab77d72cc14d880"
server_key_iv: "5ece7875b5fd9407cdb6ee4e2dc534d8"
server_key_version: "encryptionkey@0"
I20260926 20:59:53.590333 19506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:53.602308 19506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:53.604702 19506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:53.604876 19506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:53.609789 19506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43007
I20260926 20:59:53.609818 19573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43007 every 8 connection(s)
I20260926 20:59:53.610194 19506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:59:53.611078 19574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:53.613605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 19506
I20260926 20:59:53.613611 19574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26: Bootstrap starting.
I20260926 20:59:53.613727 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:59:53.613912 392 external_mini_cluster.cc:1428] Setting key 048f82d392b85caef09d5758e63ef2aa
I20260926 20:59:53.614387 19574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:53.614755 19574 log.cc:824] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:53.615499 19574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26: No bootstrap required, opened a new log
I20260926 20:59:53.617473 19574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7cdbdbfaee4875b9fcb634d6b94e26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43007 } }
I20260926 20:59:53.617621 19574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:53.617643 19574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7cdbdbfaee4875b9fcb634d6b94e26, State: Initialized, Role: FOLLOWER
I20260926 20:59:53.617801 19574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7cdbdbfaee4875b9fcb634d6b94e26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43007 } }
I20260926 20:59:53.617874 19574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:59:53.617933 19574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:59:53.617978 19574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:53.618535 19574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7cdbdbfaee4875b9fcb634d6b94e26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43007 } }
I20260926 20:59:53.618654 19574 leader_election.cc:302] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a7cdbdbfaee4875b9fcb634d6b94e26; no voters:
I20260926 20:59:53.618858 19574 leader_election.cc:288] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:59:53.618887 19579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:59:53.618997 19579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26 [term 1 LEADER]: Becoming Leader. State: Replica: 6a7cdbdbfaee4875b9fcb634d6b94e26, State: Running, Role: LEADER
I20260926 20:59:53.619078 19579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7cdbdbfaee4875b9fcb634d6b94e26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43007 } }
I20260926 20:59:53.619127 19574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:59:53.619385 19579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a7cdbdbfaee4875b9fcb634d6b94e26. Latest consensus state: current_term: 1 leader_uuid: "6a7cdbdbfaee4875b9fcb634d6b94e26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7cdbdbfaee4875b9fcb634d6b94e26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43007 } } }
I20260926 20:59:53.619458 19579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26 [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:53.619400 19580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a7cdbdbfaee4875b9fcb634d6b94e26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7cdbdbfaee4875b9fcb634d6b94e26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43007 } } }
I20260926 20:59:53.619514 19580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26 [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:53.620069 19583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:59:53.620648 19583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 20:59:53 dist-test-slave-fl7g krb5kdc[19493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:59:53.622637 19583 catalog_manager.cc:1347] Generated new cluster ID: d7c1988a512a48b0bd70742c35365052
I20260926 20:59:53.622692 19583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:59:53.627939 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:53.614508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.0.98.62:43007)
0926 20:59:53.614733 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:53.614742 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:53.614776 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:59:53.615425 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:53.615432 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:53.615486 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:53.615656 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:53.616429 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:53.617361 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:53.618353 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:53.618885 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:53.619012 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:53.622170 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:53.622174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:53.622175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:53.622189 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:53.626076 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:53.626564 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:53.626567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:53.626567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:53.626685 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:53.626948 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:53.626949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:53.626949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:53.627253 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:53.627413 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:53.627538 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 20:59:53.627681 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":120,"threads_started":1}
I20260926 20:59:53.644428 19583 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:59:53.645031 19583 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:59:53.654959 19583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26: Generated new TSK 0
I20260926 20:59:53.655189 19583 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:59:53.688205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43007
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:53.771086 19600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:53.771365 19600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:53.771412 19600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:53.772660 19600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:53.772737 19600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:53.772779 19600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:59:53.774353 19600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:53.774636 19600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:53.774875 19600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:53.780068 19606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:53.780057 19608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:53.780057 19605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:53.780359 19600 server_base.cc:1034] running on GCE node
I20260926 20:59:53.780653 19600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:53.780933 19600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:53.782120 19600 hybrid_clock.cc:648] HybridClock initialized: now 1790456393782096 us; error 35 us; skew 500 ppm
Sep 26 20:59:53 dist-test-slave-fl7g krb5kdc[19493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:53.786008 19600 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:59:53.786507 19600 webserver.cc:460] Webserver started at http://127.0.98.1:44957/ using document root <none> and password file <none>
I20260926 20:59:53.786721 19600 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:53.786767 19600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:53.786857 19600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:53.787875 19600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "e02321e83220414cb2dda467a5522030"
format_stamp: "Formatted at 2026-09-26 20:59:53 on dist-test-slave-fl7g"
server_key: "714b346178ad11e1a429532fd784b4bd"
server_key_iv: "4aebceb5c1282a717ea40b3f69189896"
server_key_version: "encryptionkey@0"
I20260926 20:59:53.788183 19600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e02321e83220414cb2dda467a5522030"
format_stamp: "Formatted at 2026-09-26 20:59:53 on dist-test-slave-fl7g"
server_key: "714b346178ad11e1a429532fd784b4bd"
server_key_iv: "4aebceb5c1282a717ea40b3f69189896"
server_key_version: "encryptionkey@0"
I20260926 20:59:53.789515 19600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:59:53.790222 19615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:53.790445 19600 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:53.790522 19600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "e02321e83220414cb2dda467a5522030"
format_stamp: "Formatted at 2026-09-26 20:59:53 on dist-test-slave-fl7g"
server_key: "714b346178ad11e1a429532fd784b4bd"
server_key_iv: "4aebceb5c1282a717ea40b3f69189896"
server_key_version: "encryptionkey@0"
I20260926 20:59:53.790617 19600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:53.825749 19600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:53.828198 19600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:53.828358 19600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:53.828666 19600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:53.828984 19600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:53.829038 19600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:53.829074 19600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:53.829102 19600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:53.834782 19600 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38733
I20260926 20:59:53.834852 19728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38733 every 8 connection(s)
I20260926 20:59:53.835177 19600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 20:59:53 dist-test-slave-fl7g krb5kdc[19493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:59:53.844110 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 19600
I20260926 20:59:53.844204 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:59:53.844472 392 external_mini_cluster.cc:1428] Setting key 5b611e4b52873bcb8e037905fdae9e97
I20260926 20:59:53.845520 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:53.836113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44003 (local address 127.0.98.62:43007)
0926 20:59:53.836180 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:53.836182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:53.836450 (+ 268us) server_negotiation.cc:407] Connection header received
0926 20:59:53.837440 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:53.837441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:53.837453 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:53.837506 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:53.838164 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:53.838669 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:53.839705 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:53.839926 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:53.840008 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:53.842406 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:53.842408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:53.842408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:53.842416 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:53.843789 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:53.844296 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:53.844298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:53.844298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:53.844354 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:53.844695 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:53.844696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:53.844696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:53.844833 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:53.844896 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:53.845281 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 20:59:53.845390 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:59:53.845582 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:53.836116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43007 (local address 127.0.98.1:44003)
0926 20:59:53.836289 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:53.836326 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:53.837324 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:53.837595 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:53.837599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:53.837872 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:53.838065 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:53.838069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:53.838776 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:53.838777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:53.839611 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:53.839613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:53.840018 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:53.840020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:53.840145 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:53.840694 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:59:53.840703 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:53.842293 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:53.843880 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:53.843886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:53.843888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:53.844124 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:53.844446 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:53.844447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:53.844448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:53.844598 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:53.844893 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:53.844900 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:53.845076 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 20:59:53.845190 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 20:59:53.845390 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260926 20:59:53.846179 19729 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43007
I20260926 20:59:53.846297 19729 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:53.846505 19729 heartbeater.cc:511] Master 127.0.98.62:43007 requested a full tablet report, sending...
I20260926 20:59:53.847149 19538 ts_manager.cc:194] Registered new tserver with Master: e02321e83220414cb2dda467a5522030 (127.0.98.1:38733)
I20260926 20:59:53.848101 19538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44003
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:59:53.871708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43007
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:53.953857 19734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:53.954079 19734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:53.954104 19734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:53.955324 19734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:53.955374 19734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:53.955401 19734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:59:53.956986 19734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:53.957254 19734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:53.957558 19734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:53.962646 19740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:53.962688 19739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:53.962636 19742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:53.962857 19734 server_base.cc:1034] running on GCE node
I20260926 20:59:53.963070 19734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:53.963330 19734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:53.964481 19734 hybrid_clock.cc:648] HybridClock initialized: now 1790456393964453 us; error 51 us; skew 500 ppm
Sep 26 20:59:53 dist-test-slave-fl7g krb5kdc[19493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:53.966975 19734 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:59:53.967442 19734 webserver.cc:460] Webserver started at http://127.0.98.2:37391/ using document root <none> and password file <none>
I20260926 20:59:53.967669 19734 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:53.967736 19734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:53.967880 19734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:53.969048 19734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "d79e47437b8841e1a1b0cf010f81b51a"
format_stamp: "Formatted at 2026-09-26 20:59:53 on dist-test-slave-fl7g"
server_key: "15cf19a5e9423e96bcfb93eacaeec779"
server_key_iv: "a05e06f310c1e949ed2d1d0c4f6e2699"
server_key_version: "encryptionkey@0"
I20260926 20:59:53.969429 19734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d79e47437b8841e1a1b0cf010f81b51a"
format_stamp: "Formatted at 2026-09-26 20:59:53 on dist-test-slave-fl7g"
server_key: "15cf19a5e9423e96bcfb93eacaeec779"
server_key_iv: "a05e06f310c1e949ed2d1d0c4f6e2699"
server_key_version: "encryptionkey@0"
I20260926 20:59:53.970749 19734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:59:53.971491 19749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:53.971662 19734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:59:53.971767 19734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "d79e47437b8841e1a1b0cf010f81b51a"
format_stamp: "Formatted at 2026-09-26 20:59:53 on dist-test-slave-fl7g"
server_key: "15cf19a5e9423e96bcfb93eacaeec779"
server_key_iv: "a05e06f310c1e949ed2d1d0c4f6e2699"
server_key_version: "encryptionkey@0"
I20260926 20:59:53.971843 19734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:53.983366 19734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:53.985903 19734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:53.986094 19734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:53.986366 19734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:53.986701 19734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:53.986758 19734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:53.986793 19734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:53.986836 19734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:53.992583 19734 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44483
I20260926 20:59:53.992622 19862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44483 every 8 connection(s)
I20260926 20:59:53.993026 19734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:59:53.996894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 19734
I20260926 20:59:53.996989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:59:53.997184 392 external_mini_cluster.cc:1428] Setting key 3fe5338fc36814bc96d1b9c0e0c4ed53
Sep 26 20:59:53 dist-test-slave-fl7g krb5kdc[19493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:59:54.003368 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:53.993829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47703 (local address 127.0.98.62:43007)
0926 20:59:53.993897 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:53.993900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:53.994199 (+ 299us) server_negotiation.cc:407] Connection header received
0926 20:59:53.995229 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:53.995230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:53.995240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:53.995305 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:53.995960 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:53.996499 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:53.997717 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:53.998034 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:53.998110 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.000318 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:54.000320 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:54.000320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:54.000328 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:54.001913 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:54.002350 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:54.002351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:54.002352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:54.002413 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:54.002690 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:54.002691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:54.002692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:54.002818 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:54.002896 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:54.003176 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 20:59:54.003272 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:59:54.003584 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:53.993840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43007 (local address 127.0.98.2:47703)
0926 20:59:53.994063 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:53.994103 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:53.995119 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:53.995362 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:53.995366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:53.995640 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:53.995874 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:53.995879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:53.996565 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:53.996566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:53.997600 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:53.997603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:53.998128 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:53.998129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:53.998277 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:53.998820 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:59:53.998828 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:54.000215 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:54.002019 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:54.002025 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:54.002028 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:54.002271 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:54.002489 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:54.002489 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:54.002490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:54.002609 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:54.002886 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:54.002893 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:54.003041 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 20:59:54.003152 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 20:59:54.003330 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":101,"threads_started":1}
I20260926 20:59:54.004169 19863 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43007
I20260926 20:59:54.004277 19863 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:54.004521 19863 heartbeater.cc:511] Master 127.0.98.62:43007 requested a full tablet report, sending...
I20260926 20:59:54.004966 19538 ts_manager.cc:194] Registered new tserver with Master: d79e47437b8841e1a1b0cf010f81b51a (127.0.98.2:44483)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 20:59:54.005514 19538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:59:54.024677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43007
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:54.108124 19868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:54.108346 19868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:54.108371 19868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:54.109642 19868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:54.109683 19868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:54.109709 19868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:59:54.111155 19868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:54.111341 19868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:54.111531 19868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:54.116500 19876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:54.116477 19873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:54.116593 19874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:54.116876 19868 server_base.cc:1034] running on GCE node
I20260926 20:59:54.117110 19868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:54.117358 19868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:54.118520 19868 hybrid_clock.cc:648] HybridClock initialized: now 1790456394118511 us; error 29 us; skew 500 ppm
Sep 26 20:59:54 dist-test-slave-fl7g krb5kdc[19493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:54.121465 19868 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:59:54.121938 19868 webserver.cc:460] Webserver started at http://127.0.98.3:38507/ using document root <none> and password file <none>
I20260926 20:59:54.122176 19868 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:54.122243 19868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:54.122385 19868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:54.123437 19868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "54d6f3e4bb9b406ebe3610b326bfe7db"
format_stamp: "Formatted at 2026-09-26 20:59:54 on dist-test-slave-fl7g"
server_key: "4583842ea728dd5c7c7205274b80f95c"
server_key_iv: "91cd468a8f8bc6eb1d6dda81289b2e17"
server_key_version: "encryptionkey@0"
I20260926 20:59:54.123821 19868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "54d6f3e4bb9b406ebe3610b326bfe7db"
format_stamp: "Formatted at 2026-09-26 20:59:54 on dist-test-slave-fl7g"
server_key: "4583842ea728dd5c7c7205274b80f95c"
server_key_iv: "91cd468a8f8bc6eb1d6dda81289b2e17"
server_key_version: "encryptionkey@0"
I20260926 20:59:54.125233 19868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 20:59:54.126032 19883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:54.126251 19868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:54.126370 19868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "54d6f3e4bb9b406ebe3610b326bfe7db"
format_stamp: "Formatted at 2026-09-26 20:59:54 on dist-test-slave-fl7g"
server_key: "4583842ea728dd5c7c7205274b80f95c"
server_key_iv: "91cd468a8f8bc6eb1d6dda81289b2e17"
server_key_version: "encryptionkey@0"
I20260926 20:59:54.126444 19868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:54.174757 19868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:54.177480 19868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:54.177660 19868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:54.177949 19868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:54.178315 19868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:54.178401 19868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:54.178448 19868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:54.178479 19868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:54.184571 19868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42171
I20260926 20:59:54.184610 19996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42171 every 8 connection(s)
I20260926 20:59:54.184957 19868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:59:54.190459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 19868
I20260926 20:59:54.190565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:59:54.190801 392 external_mini_cluster.cc:1428] Setting key 6fa9ae048d02f77656582f0d61aad376
Sep 26 20:59:54 dist-test-slave-fl7g krb5kdc[19493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:59:54.195030 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.185770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34929 (local address 127.0.98.62:43007)
0926 20:59:54.185832 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:54.185834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:54.186043 (+ 209us) server_negotiation.cc:407] Connection header received
0926 20:59:54.187030 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:54.187031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:54.187042 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:54.187105 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:54.187720 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.188232 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.189301 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.189495 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.189557 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.192044 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:54.192045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:54.192046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:54.192054 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:54.193558 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:54.194002 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:54.194003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:54.194003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:54.194069 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:54.194372 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:54.194373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:54.194374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:54.194493 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:54.194573 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:54.194855 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 20:59:54.194939 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:59:54.195242 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.185734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43007 (local address 127.0.98.3:34929)
0926 20:59:54.185910 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:54.185956 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:54.186935 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:54.187162 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:54.187165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:54.187435 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:54.187633 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.187637 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.188329 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.188330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.189217 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.189219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.189576 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.189577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.189678 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.190180 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:59:54.190189 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:54.191963 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:54.193654 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:54.193661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:54.193663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:54.193909 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:54.194162 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:59:54.194163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:59:54.194164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:59:54.194276 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:59:54.194563 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:54.194570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:54.194718 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 20:59:54.194825 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 20:59:54.194989 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":105,"threads_started":1}
I20260926 20:59:54.195736 19997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43007
I20260926 20:59:54.195827 19997 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:54.195971 19997 heartbeater.cc:511] Master 127.0.98.62:43007 requested a full tablet report, sending...
I20260926 20:59:54.196331 19538 ts_manager.cc:194] Registered new tserver with Master: 54d6f3e4bb9b406ebe3610b326bfe7db (127.0.98.3:42171)
I20260926 20:59:54.196764 19538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34929
I20260926 20:59:54.202971 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:59:54.210513 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.204340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.0.98.62:43007)
0926 20:59:54.204379 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:54.204380 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:54.204511 (+ 131us) server_negotiation.cc:407] Connection header received
0926 20:59:54.204565 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:54.204565 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:54.204574 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:54.204625 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:54.205210 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.205810 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.206534 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.206728 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.206795 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.208118 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:54.208119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:54.208120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:54.208126 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:54.209524 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:54.209871 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:54.209872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:54.209872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:54.209922 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:54.210081 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:54.210081 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:54.210081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:54.210234 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:54.210312 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:54.210393 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 20:59:54.210450 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:59:54.212502 19538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:59:54.214141 19538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:59:54.221001 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.216582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42171 (local address 127.0.0.1:33582)
0926 20:59:54.216746 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:54.216789 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:54.216852 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:54.217341 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:54.217344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:54.217348 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:54.217424 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.217426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.218846 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.218847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.220007 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.220009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.220730 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.220731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.220802 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.220806 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:59:54.220865 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 20:59:54.220914 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":53,"threads_started":1}
I20260926 20:59:54.221009 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.216594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44483 (local address 127.0.0.1:54432)
0926 20:59:54.217118 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:54.217130 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:54.217189 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:54.217381 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:54.217382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:54.217384 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:54.217443 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.217445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.218440 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.218442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.219652 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.219655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.220650 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.220651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.220731 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.220746 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 20:59:54.220838 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:59:54.220904 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":97,"threads_started":1}
I20260926 20:59:54.221254 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.216731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.0.98.2:44483)
0926 20:59:54.217186 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:54.217191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:54.217202 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:59:54.217259 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:54.217261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:54.217279 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:54.217519 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:54.217596 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.218336 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.219739 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.220513 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.220579 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.221064 (+ 485us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:54.221090 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 20:59:54.221139 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":108,"threads_started":1}
I20260926 20:59:54.221076 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.216942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33582 (local address 127.0.98.3:42171)
0926 20:59:54.217147 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:54.217152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:54.217165 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:59:54.217204 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:54.217207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:54.217222 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:54.217505 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:54.217594 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.218730 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.220090 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.220635 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.220727 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.220790 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:54.220910 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 20:59:54.220983 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
I20260926 20:59:54.221472 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.216823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38733 (local address 127.0.0.1:54248)
0926 20:59:54.217307 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:54.217315 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:54.217361 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:54.217638 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:54.217639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:54.217641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:54.217680 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.217681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.218929 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.218930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.220031 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.220033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.221000 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.221000 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.221055 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.221058 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:59:54.221215 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 20:59:54.221394 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":80,"threads_started":1}
I20260926 20:59:54.221771 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.216984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54248 (local address 127.0.98.1:38733)
0926 20:59:54.217159 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:54.217163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:54.217419 (+ 256us) server_negotiation.cc:407] Connection header received
0926 20:59:54.217462 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:54.217465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:54.217481 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:54.217549 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:54.217762 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.218847 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.220121 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.220859 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.220916 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.220968 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:54.221625 (+ 657us) server_negotiation.cc:299] Negotiation successful
0926 20:59:54.221670 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260926 20:59:54.222034 19797 tablet_service.cc:1467] Processing CreateTablet for tablet 9e0c29f43e2f4b3487d1dfcc8be00dd6 (DEFAULT_TABLE table=test-table [id=e7446cf23bf94db7b985a01c3a20bcb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:54.222033 19931 tablet_service.cc:1467] Processing CreateTablet for tablet 9e0c29f43e2f4b3487d1dfcc8be00dd6 (DEFAULT_TABLE table=test-table [id=e7446cf23bf94db7b985a01c3a20bcb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:54.222414 19797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e0c29f43e2f4b3487d1dfcc8be00dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:54.222414 19931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e0c29f43e2f4b3487d1dfcc8be00dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:54.225893 20015 tablet_bootstrap.cc:492] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a: Bootstrap starting.
I20260926 20:59:54.226130 20014 tablet_bootstrap.cc:492] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db: Bootstrap starting.
I20260926 20:59:54.226297 19663 tablet_service.cc:1467] Processing CreateTablet for tablet 9e0c29f43e2f4b3487d1dfcc8be00dd6 (DEFAULT_TABLE table=test-table [id=e7446cf23bf94db7b985a01c3a20bcb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:54.226627 19663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e0c29f43e2f4b3487d1dfcc8be00dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:54.226812 20014 tablet_bootstrap.cc:654] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:54.227336 20014 log.cc:824] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:54.228116 20014 tablet_bootstrap.cc:492] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db: No bootstrap required, opened a new log
I20260926 20:59:54.228193 20014 ts_tablet_manager.cc:1397] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:59:54.228201 20015 tablet_bootstrap.cc:654] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:54.228754 20015 log.cc:824] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:54.229475 20015 tablet_bootstrap.cc:492] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a: No bootstrap required, opened a new log
I20260926 20:59:54.229557 20015 ts_tablet_manager.cc:1397] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 20:59:54.229879 20018 tablet_bootstrap.cc:492] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030: Bootstrap starting.
I20260926 20:59:54.230337 20014 raft_consensus.cc:348] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d6f3e4bb9b406ebe3610b326bfe7db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } } peers { permanent_uuid: "d79e47437b8841e1a1b0cf010f81b51a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44483 } } peers { permanent_uuid: "e02321e83220414cb2dda467a5522030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38733 } }
I20260926 20:59:54.230504 20014 raft_consensus.cc:374] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:54.230548 20014 raft_consensus.cc:729] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54d6f3e4bb9b406ebe3610b326bfe7db, State: Initialized, Role: FOLLOWER
I20260926 20:59:54.230619 20018 tablet_bootstrap.cc:654] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:54.230657 20014 consensus_queue.cc:260] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d6f3e4bb9b406ebe3610b326bfe7db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } } peers { permanent_uuid: "d79e47437b8841e1a1b0cf010f81b51a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44483 } } peers { permanent_uuid: "e02321e83220414cb2dda467a5522030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38733 } }
I20260926 20:59:54.230930 20014 ts_tablet_manager.cc:1428] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:59:54.231062 19997 heartbeater.cc:503] Master 127.0.98.62:43007 was elected leader, sending a full tablet report...
I20260926 20:59:54.231160 20018 log.cc:824] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:54.231822 20015 raft_consensus.cc:348] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d6f3e4bb9b406ebe3610b326bfe7db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } } peers { permanent_uuid: "d79e47437b8841e1a1b0cf010f81b51a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44483 } } peers { permanent_uuid: "e02321e83220414cb2dda467a5522030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38733 } }
I20260926 20:59:54.232013 20015 raft_consensus.cc:374] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:54.232056 20015 raft_consensus.cc:729] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d79e47437b8841e1a1b0cf010f81b51a, State: Initialized, Role: FOLLOWER
I20260926 20:59:54.232180 20015 consensus_queue.cc:260] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d6f3e4bb9b406ebe3610b326bfe7db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } } peers { permanent_uuid: "d79e47437b8841e1a1b0cf010f81b51a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44483 } } peers { permanent_uuid: "e02321e83220414cb2dda467a5522030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38733 } }
I20260926 20:59:54.232520 19863 heartbeater.cc:503] Master 127.0.98.62:43007 was elected leader, sending a full tablet report...
I20260926 20:59:54.232518 20015 ts_tablet_manager.cc:1428] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:59:54.233007 20018 tablet_bootstrap.cc:492] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030: No bootstrap required, opened a new log
I20260926 20:59:54.233081 20018 ts_tablet_manager.cc:1397] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:59:54.234644 20018 raft_consensus.cc:348] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d6f3e4bb9b406ebe3610b326bfe7db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } } peers { permanent_uuid: "d79e47437b8841e1a1b0cf010f81b51a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44483 } } peers { permanent_uuid: "e02321e83220414cb2dda467a5522030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38733 } }
I20260926 20:59:54.234777 20018 raft_consensus.cc:374] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:54.234800 20018 raft_consensus.cc:729] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e02321e83220414cb2dda467a5522030, State: Initialized, Role: FOLLOWER
I20260926 20:59:54.234929 20018 consensus_queue.cc:260] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d6f3e4bb9b406ebe3610b326bfe7db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } } peers { permanent_uuid: "d79e47437b8841e1a1b0cf010f81b51a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44483 } } peers { permanent_uuid: "e02321e83220414cb2dda467a5522030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38733 } }
I20260926 20:59:54.235208 20018 ts_tablet_manager.cc:1428] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:59:54.235239 19729 heartbeater.cc:503] Master 127.0.98.62:43007 was elected leader, sending a full tablet report...
W20260926 20:59:54.243643 19864 tablet.cc:2369] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:59:54.257392 20022 raft_consensus.cc:482] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:59:54.257565 20022 raft_consensus.cc:504] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d6f3e4bb9b406ebe3610b326bfe7db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } } peers { permanent_uuid: "d79e47437b8841e1a1b0cf010f81b51a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44483 } } peers { permanent_uuid: "e02321e83220414cb2dda467a5522030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38733 } }
I20260926 20:59:54.257978 20022 leader_election.cc:288] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54d6f3e4bb9b406ebe3610b326bfe7db (127.0.98.3:42171), d79e47437b8841e1a1b0cf010f81b51a (127.0.98.2:44483)
I20260926 20:59:54.262023 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.258276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44193 (local address 127.0.98.3:42171)
0926 20:59:54.258360 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:54.258362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:54.258373 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:59:54.258406 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:54.258407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:54.258419 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:54.258494 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:54.258715 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.259488 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.261027 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.261615 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.261691 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.261701 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:54.261911 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 20:59:54.261955 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:59:54.262033 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.258148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42171 (local address 127.0.98.1:44193)
0926 20:59:54.258212 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:54.258225 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:54.258287 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:54.258526 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:54.258527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:54.258529 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:54.258620 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.258621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.259585 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.259586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.260921 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.260925 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.261739 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.261740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.261839 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.261843 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:59:54.261893 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 20:59:54.261938 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":3}
I20260926 20:59:54.262122 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.258363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43577 (local address 127.0.98.2:44483)
0926 20:59:54.258416 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:54.258418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:54.258480 (+ 62us) server_negotiation.cc:407] Connection header received
0926 20:59:54.258527 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:54.258528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:54.258540 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:54.258594 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:54.258784 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.259487 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.261026 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.261615 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.261693 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.261701 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:54.261995 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 20:59:54.262049 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 20:59:54.262045 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.258148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44483 (local address 127.0.98.1:43577)
0926 20:59:54.258409 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:54.258425 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:54.258465 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:54.258614 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:54.258615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:54.258617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:54.258711 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.258713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.259585 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.259586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.260921 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.260925 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.261727 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.261729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.261811 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.261816 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:59:54.261891 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:59:54.261932 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":115,"thread_start_us":137,"threads_started":1}
I20260926 20:59:54.262423 19951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0c29f43e2f4b3487d1dfcc8be00dd6" candidate_uuid: "e02321e83220414cb2dda467a5522030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54d6f3e4bb9b406ebe3610b326bfe7db" is_pre_election: true
I20260926 20:59:54.262605 19951 raft_consensus.cc:2454] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e02321e83220414cb2dda467a5522030 in term 0.
I20260926 20:59:54.262799 19817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0c29f43e2f4b3487d1dfcc8be00dd6" candidate_uuid: "e02321e83220414cb2dda467a5522030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79e47437b8841e1a1b0cf010f81b51a" is_pre_election: true
I20260926 20:59:54.262852 19619 leader_election.cc:302] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54d6f3e4bb9b406ebe3610b326bfe7db, e02321e83220414cb2dda467a5522030; no voters:
I20260926 20:59:54.262953 19817 raft_consensus.cc:2454] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e02321e83220414cb2dda467a5522030 in term 0.
I20260926 20:59:54.263016 20022 raft_consensus.cc:2781] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:59:54.263074 20022 raft_consensus.cc:482] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:59:54.263098 20022 raft_consensus.cc:3037] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:54.263801 20022 raft_consensus.cc:504] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d6f3e4bb9b406ebe3610b326bfe7db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } } peers { permanent_uuid: "d79e47437b8841e1a1b0cf010f81b51a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44483 } } peers { permanent_uuid: "e02321e83220414cb2dda467a5522030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38733 } }
I20260926 20:59:54.263949 20022 leader_election.cc:288] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [CANDIDATE]: Term 1 election: Requested vote from peers 54d6f3e4bb9b406ebe3610b326bfe7db (127.0.98.3:42171), d79e47437b8841e1a1b0cf010f81b51a (127.0.98.2:44483)
I20260926 20:59:54.264149 19951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0c29f43e2f4b3487d1dfcc8be00dd6" candidate_uuid: "e02321e83220414cb2dda467a5522030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54d6f3e4bb9b406ebe3610b326bfe7db"
I20260926 20:59:54.264168 19817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0c29f43e2f4b3487d1dfcc8be00dd6" candidate_uuid: "e02321e83220414cb2dda467a5522030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79e47437b8841e1a1b0cf010f81b51a"
I20260926 20:59:54.264254 19817 raft_consensus.cc:3037] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:54.264262 19951 raft_consensus.cc:3037] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:54.265007 19817 raft_consensus.cc:2454] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e02321e83220414cb2dda467a5522030 in term 1.
I20260926 20:59:54.265169 19951 raft_consensus.cc:2454] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e02321e83220414cb2dda467a5522030 in term 1.
I20260926 20:59:54.265203 19617 leader_election.cc:302] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d79e47437b8841e1a1b0cf010f81b51a, e02321e83220414cb2dda467a5522030; no voters:
I20260926 20:59:54.265321 20022 raft_consensus.cc:2781] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:59:54.265501 20022 raft_consensus.cc:686] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [term 1 LEADER]: Becoming Leader. State: Replica: e02321e83220414cb2dda467a5522030, State: Running, Role: LEADER
I20260926 20:59:54.265601 20022 consensus_queue.cc:237] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d6f3e4bb9b406ebe3610b326bfe7db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } } peers { permanent_uuid: "d79e47437b8841e1a1b0cf010f81b51a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44483 } } peers { permanent_uuid: "e02321e83220414cb2dda467a5522030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38733 } }
I20260926 20:59:54.266520 19538 catalog_manager.cc:5495] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 reported cstate change: term changed from 0 to 1, leader changed from <none> to e02321e83220414cb2dda467a5522030 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e02321e83220414cb2dda467a5522030" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d6f3e4bb9b406ebe3610b326bfe7db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d79e47437b8841e1a1b0cf010f81b51a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e02321e83220414cb2dda467a5522030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38733 } health_report { overall_health: HEALTHY } } }
I20260926 20:59:54.279222 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.276745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54264 (local address 127.0.98.1:38733)
0926 20:59:54.276836 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:54.276838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:54.276850 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:59:54.276887 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:54.276888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:54.276909 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:54.276987 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:59:54.277152 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.277654 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.278443 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.278649 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.278716 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.278912 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:59:54.279018 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:59:54.279083 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:54.279119 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 20:59:54.279139 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:59:54.281430 19817 raft_consensus.cc:1261] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a [term 1 FOLLOWER]: Refusing update from remote peer e02321e83220414cb2dda467a5522030: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:59:54.281430 19951 raft_consensus.cc:1261] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db [term 1 FOLLOWER]: Refusing update from remote peer e02321e83220414cb2dda467a5522030: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:59:54.281843 20024 consensus_queue.cc:1035] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54d6f3e4bb9b406ebe3610b326bfe7db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:59:54.282032 20022 consensus_queue.cc:1035] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d79e47437b8841e1a1b0cf010f81b51a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:59:54.284617 20027 mvcc.cc:205] Tried to move back new op lower bound from 7333709390973026304 to 7333709390912606208. Current Snapshot: MvccSnapshot[applied={T|T < 7333709390973026304}]
I20260926 20:59:54.284729 20029 mvcc.cc:205] Tried to move back new op lower bound from 7333709390973026304 to 7333709390912606208. Current Snapshot: MvccSnapshot[applied={T|T < 7333709390973026304}]
I20260926 20:59:54.285714 20028 mvcc.cc:205] Tried to move back new op lower bound from 7333709390973026304 to 7333709390912606208. Current Snapshot: MvccSnapshot[applied={T|T < 7333709390973026304 or (T in {7333709390973026304})}]
I20260926 20:59:54.287185 19538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:59:54.287281 19538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:59:54.288046 19538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a7cdbdbfaee4875b9fcb634d6b94e26: Sending DeleteTablet for 3 replicas of tablet 9e0c29f43e2f4b3487d1dfcc8be00dd6
I20260926 20:59:54.288466 19797 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e0c29f43e2f4b3487d1dfcc8be00dd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54432
I20260926 20:59:54.288475 19663 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e0c29f43e2f4b3487d1dfcc8be00dd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54248
I20260926 20:59:54.288733 20037 tablet_replica.cc:330] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a: stopping tablet replica
I20260926 20:59:54.288759 20038 tablet_replica.cc:330] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030: stopping tablet replica
I20260926 20:59:54.288851 20037 raft_consensus.cc:2229] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:59:54.288923 20038 raft_consensus.cc:2229] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:59:54.288982 20037 raft_consensus.cc:2258] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:59:54.288959 19931 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e0c29f43e2f4b3487d1dfcc8be00dd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33582
I20260926 20:59:54.289059 20038 raft_consensus.cc:2258] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P e02321e83220414cb2dda467a5522030 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:59:54.289435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 19600
I20260926 20:59:54.289520 20037 ts_tablet_manager.cc:1905] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:59:54.291359 20039 tablet_replica.cc:330] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db: stopping tablet replica
I20260926 20:59:54.291494 20039 raft_consensus.cc:2229] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:59:54.291611 20039 raft_consensus.cc:2258] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:59:54.292001 20037 ts_tablet_manager.cc:1918] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:59:54.292035 20039 ts_tablet_manager.cc:1905] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:59:54.292093 20037 log.cc:1196] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/9e0c29f43e2f4b3487d1dfcc8be00dd6
I20260926 20:59:54.292490 20037 ts_tablet_manager.cc:1939] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P d79e47437b8841e1a1b0cf010f81b51a: Deleting consensus metadata
I20260926 20:59:54.293016 19523 catalog_manager.cc:4841] TS d79e47437b8841e1a1b0cf010f81b51a (127.0.98.2:44483): tablet 9e0c29f43e2f4b3487d1dfcc8be00dd6 (table test-table [id=e7446cf23bf94db7b985a01c3a20bcb8]) successfully deleted
I20260926 20:59:54.293560 20039 ts_tablet_manager.cc:1918] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:59:54.293649 20039 log.cc:1196] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/9e0c29f43e2f4b3487d1dfcc8be00dd6
I20260926 20:59:54.293905 20039 ts_tablet_manager.cc:1939] T 9e0c29f43e2f4b3487d1dfcc8be00dd6 P 54d6f3e4bb9b406ebe3610b326bfe7db: Deleting consensus metadata
I20260926 20:59:54.294312 19525 catalog_manager.cc:4841] TS 54d6f3e4bb9b406ebe3610b326bfe7db (127.0.98.3:42171): tablet 9e0c29f43e2f4b3487d1dfcc8be00dd6 (table test-table [id=e7446cf23bf94db7b985a01c3a20bcb8]) successfully deleted
W20260926 20:59:54.295222 19525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38733 (error 108)
I20260926 20:59:54.295432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 19734
I20260926 20:59:54.296518 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.295628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38733 (local address 127.0.0.1:54266)
0926 20:59:54.296378 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:54.296477 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":709}
W20260926 20:59:54.296672 19525 catalog_manager.cc:4584] TS e02321e83220414cb2dda467a5522030 (127.0.98.1:38733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e0c29f43e2f4b3487d1dfcc8be00dd6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38733: connect: Connection refused (error 111)
I20260926 20:59:54.300736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 19868
I20260926 20:59:54.305332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 19506
2026-09-26T20:59:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1987 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T20:59:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:59:54Z Disabled control of system clock
I20260926 20:59:54.332934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38971
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:59:54.415691 20044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:54.415953 20044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:54.415977 20044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:54.417423 20044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:59:54.417480 20044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:54.417493 20044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:54.417507 20044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:59:54.417519 20044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:59:54.418938 20044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:54.419157 20044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:54.419416 20044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:54.424705 20050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:54.424690 20052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:54.424690 20049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:54.424780 20044 server_base.cc:1034] running on GCE node
I20260926 20:59:54.425334 20044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:54.425652 20044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:54.426846 20044 hybrid_clock.cc:648] HybridClock initialized: now 1790456394426809 us; error 56 us; skew 500 ppm
I20260926 20:59:54.428224 20044 webserver.cc:460] Webserver started at http://127.0.98.62:39113/ using document root <none> and password file <none>
I20260926 20:59:54.428536 20044 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:54.428611 20044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:54.428779 20044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:54.429932 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "6d5a620f462a4d4a93f8d76374912e4d"
format_stamp: "Formatted at 2026-09-26 20:59:54 on dist-test-slave-fl7g"
server_key: "64aa56810f73bdc573bbd889e150ec93"
server_key_iv: "60979d2c78c6882451774b6a81e71d46"
server_key_version: "encryptionkey@0"
I20260926 20:59:54.430335 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "6d5a620f462a4d4a93f8d76374912e4d"
format_stamp: "Formatted at 2026-09-26 20:59:54 on dist-test-slave-fl7g"
server_key: "64aa56810f73bdc573bbd889e150ec93"
server_key_iv: "60979d2c78c6882451774b6a81e71d46"
server_key_version: "encryptionkey@0"
I20260926 20:59:54.431754 20044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:59:54.432652 20058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:54.432859 20044 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:54.432971 20044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "6d5a620f462a4d4a93f8d76374912e4d"
format_stamp: "Formatted at 2026-09-26 20:59:54 on dist-test-slave-fl7g"
server_key: "64aa56810f73bdc573bbd889e150ec93"
server_key_iv: "60979d2c78c6882451774b6a81e71d46"
server_key_version: "encryptionkey@0"
I20260926 20:59:54.433054 20044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:54.447301 20044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:54.447676 20044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:54.447854 20044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:54.451825 20044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38971
I20260926 20:59:54.451855 20110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38971 every 8 connection(s)
I20260926 20:59:54.452240 20044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:59:54.453317 20111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:54.455753 20111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d: Bootstrap starting.
I20260926 20:59:54.456401 20111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:54.456805 20111 log.cc:824] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:54.457495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 20044
I20260926 20:59:54.457609 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:59:54.457684 20111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d: No bootstrap required, opened a new log
I20260926 20:59:54.457803 392 external_mini_cluster.cc:1428] Setting key 4e807cab255997ef5991f2a3cb7ac6b9
I20260926 20:59:54.460489 20111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5a620f462a4d4a93f8d76374912e4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38971 } }
I20260926 20:59:54.460695 20111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:54.460737 20111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5a620f462a4d4a93f8d76374912e4d, State: Initialized, Role: FOLLOWER
I20260926 20:59:54.460883 20111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5a620f462a4d4a93f8d76374912e4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38971 } }
I20260926 20:59:54.460969 20111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:59:54.461001 20111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:59:54.461047 20111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:54.461805 20111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5a620f462a4d4a93f8d76374912e4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38971 } }
I20260926 20:59:54.461934 20111 leader_election.cc:302] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d5a620f462a4d4a93f8d76374912e4d; no voters:
I20260926 20:59:54.462142 20111 leader_election.cc:288] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:59:54.462173 20116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:59:54.462292 20116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d [term 1 LEADER]: Becoming Leader. State: Replica: 6d5a620f462a4d4a93f8d76374912e4d, State: Running, Role: LEADER
I20260926 20:59:54.462378 20116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5a620f462a4d4a93f8d76374912e4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38971 } }
I20260926 20:59:54.462447 20111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:59:54.462679 20116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d5a620f462a4d4a93f8d76374912e4d. Latest consensus state: current_term: 1 leader_uuid: "6d5a620f462a4d4a93f8d76374912e4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5a620f462a4d4a93f8d76374912e4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38971 } } }
I20260926 20:59:54.462697 20117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d5a620f462a4d4a93f8d76374912e4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5a620f462a4d4a93f8d76374912e4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38971 } } }
I20260926 20:59:54.462746 20116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:54.462776 20117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:54.463049 20120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:59:54.463608 20120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:59:54.463907 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.458398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49528 (local address 127.0.98.62:38971)
0926 20:59:54.458618 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:54.458626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:54.458651 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:59:54.459131 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:54.459138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:54.459209 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:54.459336 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:54.459581 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.460547 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.461647 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.462159 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.462708 (+ 549us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.463102 (+ 394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:54.463107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:54.463108 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:54.463340 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:54.463389 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:54.463405 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:54.463484 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:54.463576 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 20:59:54.463690 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":135,"thread_start_us":102,"threads_started":1}
I20260926 20:59:54.465864 20120 catalog_manager.cc:1347] Generated new cluster ID: e5363f3f450f45b0a9e4f9ac59b16add
I20260926 20:59:54.465924 20120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:59:54.475046 20120 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:59:54.475816 20120 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:59:54.482801 20120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d: Generated new TSK 0
I20260926 20:59:54.483044 20120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:59:54.485459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38971
--builtin_ntp_servers=127.0.98.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:59:54.570086 20135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:54.570335 20135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:54.570375 20135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:54.571605 20135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:54.571678 20135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:54.571724 20135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:59:54.573215 20135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:54.573485 20135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:54.573803 20135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:54.579969 20140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:54.580000 20141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:54.579985 20143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:54.580173 20135 server_base.cc:1034] running on GCE node
I20260926 20:59:54.580564 20135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:54.580832 20135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:54.582017 20135 hybrid_clock.cc:648] HybridClock initialized: now 1790456394581992 us; error 46 us; skew 500 ppm
I20260926 20:59:54.583408 20135 webserver.cc:460] Webserver started at http://127.0.98.1:36215/ using document root <none> and password file <none>
I20260926 20:59:54.583665 20135 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:54.583735 20135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:54.583910 20135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:54.585107 20135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-0/data/instance:
uuid: "87ecf16c37c24a6b8570d48b720e6f06"
format_stamp: "Formatted at 2026-09-26 20:59:54 on dist-test-slave-fl7g"
server_key: "de91406dc1cf1b2179e361e4afd5f0eb"
server_key_iv: "c547185257bdc530a12de6ac5731eb31"
server_key_version: "encryptionkey@0"
I20260926 20:59:54.585500 20135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-0/wal/instance:
uuid: "87ecf16c37c24a6b8570d48b720e6f06"
format_stamp: "Formatted at 2026-09-26 20:59:54 on dist-test-slave-fl7g"
server_key: "de91406dc1cf1b2179e361e4afd5f0eb"
server_key_iv: "c547185257bdc530a12de6ac5731eb31"
server_key_version: "encryptionkey@0"
I20260926 20:59:54.586969 20135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:59:54.587817 20149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:54.588043 20135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:54.588157 20135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-0/wal
uuid: "87ecf16c37c24a6b8570d48b720e6f06"
format_stamp: "Formatted at 2026-09-26 20:59:54 on dist-test-slave-fl7g"
server_key: "de91406dc1cf1b2179e361e4afd5f0eb"
server_key_iv: "c547185257bdc530a12de6ac5731eb31"
server_key_version: "encryptionkey@0"
I20260926 20:59:54.588256 20135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:54.595702 20135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:54.596021 20135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:54.596168 20135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:54.596441 20135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:54.596792 20135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:54.596846 20135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:54.596889 20135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:54.596915 20135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:54.602747 20135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42893
I20260926 20:59:54.602818 20262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42893 every 8 connection(s)
I20260926 20:59:54.603168 20135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:59:54.608843 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.604018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39939 (local address 127.0.98.62:38971)
0926 20:59:54.604103 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:54.604105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:54.604319 (+ 214us) server_negotiation.cc:407] Connection header received
0926 20:59:54.604939 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:54.604940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:54.604957 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:54.605006 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:54.605734 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.606240 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.607321 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.607522 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.607601 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.608554 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:54.608556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:54.608556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:54.608561 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:54.608576 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:54.608582 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:54.608648 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:54.608733 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 20:59:54.608760 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:59:54.609071 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.603962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38971 (local address 127.0.98.1:39939)
0926 20:59:54.604196 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:54.604231 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:54.604823 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:54.605091 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:54.605095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:54.605112 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:54.605627 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.605632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.606360 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.606362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.607208 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.607210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.607609 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.607610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.607755 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.608340 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:59:54.608340 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:54.608371 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:59:54.608395 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:54.608648 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:54.608655 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:54.608658 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:59:54.608735 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:59:54.608840 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
I20260926 20:59:54.609719 20263 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38971
I20260926 20:59:54.609812 20263 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:54.609959 20263 heartbeater.cc:511] Master 127.0.98.62:38971 requested a full tablet report, sending...
I20260926 20:59:54.610466 20075 ts_manager.cc:194] Registered new tserver with Master: 87ecf16c37c24a6b8570d48b720e6f06 (127.0.98.1:42893)
I20260926 20:59:54.611045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 20135
I20260926 20:59:54.611227 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:59:54.611200 20075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:39939
I20260926 20:59:54.611537 392 external_mini_cluster.cc:1428] Setting key f4bb6a47ebe5310b53c94bce85ffdac1
I20260926 20:59:54.612668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38971
--builtin_ntp_servers=127.0.98.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:59:54.694360 20266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:54.694614 20266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:54.694664 20266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:54.695895 20266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:54.695964 20266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:54.696009 20266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:59:54.697464 20266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:54.697742 20266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:54.698045 20266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:54.703650 20272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:54.703775 20266 server_base.cc:1034] running on GCE node
W20260926 20:59:54.703688 20271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:54.703650 20274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:54.704062 20266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:54.704274 20266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:54.705425 20266 hybrid_clock.cc:648] HybridClock initialized: now 1790456394705410 us; error 35 us; skew 500 ppm
I20260926 20:59:54.706753 20266 webserver.cc:460] Webserver started at http://127.0.98.2:33027/ using document root <none> and password file <none>
I20260926 20:59:54.707001 20266 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:54.707083 20266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:54.707198 20266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:54.708233 20266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-1/data/instance:
uuid: "95958c41aecf40d7a4ca638a39f2aea1"
format_stamp: "Formatted at 2026-09-26 20:59:54 on dist-test-slave-fl7g"
server_key: "e1033f0fd2626c17ab82eef6fc791758"
server_key_iv: "b8fbaad4791e4d76d64360e80d5b68d3"
server_key_version: "encryptionkey@0"
I20260926 20:59:54.708621 20266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-1/wal/instance:
uuid: "95958c41aecf40d7a4ca638a39f2aea1"
format_stamp: "Formatted at 2026-09-26 20:59:54 on dist-test-slave-fl7g"
server_key: "e1033f0fd2626c17ab82eef6fc791758"
server_key_iv: "b8fbaad4791e4d76d64360e80d5b68d3"
server_key_version: "encryptionkey@0"
I20260926 20:59:54.709914 20266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:59:54.710646 20280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:54.710827 20266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:59:54.710896 20266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-1/wal
uuid: "95958c41aecf40d7a4ca638a39f2aea1"
format_stamp: "Formatted at 2026-09-26 20:59:54 on dist-test-slave-fl7g"
server_key: "e1033f0fd2626c17ab82eef6fc791758"
server_key_iv: "b8fbaad4791e4d76d64360e80d5b68d3"
server_key_version: "encryptionkey@0"
I20260926 20:59:54.710949 20266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:54.735183 20266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:54.735479 20266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:54.735586 20266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:54.735874 20266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:54.736186 20266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:54.736217 20266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:54.736243 20266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:54.736254 20266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:54.742208 20266 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35723
I20260926 20:59:54.742301 20393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35723 every 8 connection(s)
I20260926 20:59:54.742640 20266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:59:54.747808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 20266
I20260926 20:59:54.747929 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:59:54.748063 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.743515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33861 (local address 127.0.98.62:38971)
0926 20:59:54.743586 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:54.743588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:54.743775 (+ 187us) server_negotiation.cc:407] Connection header received
0926 20:59:54.744255 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:54.744257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:54.744267 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:54.744334 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:54.745070 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.745562 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.746576 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.746809 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.746875 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.747751 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:54.747753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:54.747753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:54.747759 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:54.747778 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:54.747783 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:54.747849 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:54.747951 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 20:59:54.747981 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:59:54.748165 392 external_mini_cluster.cc:1428] Setting key cb291525f848463d81a8c4dcd6533d72
I20260926 20:59:54.748299 20396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.743402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38971 (local address 127.0.98.2:33861)
0926 20:59:54.743603 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:54.743646 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:54.744160 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:54.744457 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:54.744461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:54.744473 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:54.744991 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.744996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.745643 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.745644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.746438 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.746440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.746896 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.746897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.747006 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.747592 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:59:54.747592 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:54.747622 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:59:54.747645 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:54.747854 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:54.747859 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:54.747861 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:59:54.747950 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:59:54.748053 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260926 20:59:54.748843 20394 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38971
I20260926 20:59:54.748953 20394 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:54.749181 20394 heartbeater.cc:511] Master 127.0.98.62:38971 requested a full tablet report, sending...
I20260926 20:59:54.749213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38971
--builtin_ntp_servers=127.0.98.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:59:54.749544 20075 ts_manager.cc:194] Registered new tserver with Master: 95958c41aecf40d7a4ca638a39f2aea1 (127.0.98.2:35723)
I20260926 20:59:54.749950 20075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:33861
W20260926 20:59:54.832823 20397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:54.833091 20397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:54.833128 20397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:54.834378 20397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:54.834444 20397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:54.834486 20397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:59:54.835968 20397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:54.836227 20397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:54.836596 20397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:54.841792 20402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:54.841809 20403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:54.841938 20397 server_base.cc:1034] running on GCE node
W20260926 20:59:54.841882 20405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:54.842365 20397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:54.842612 20397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:54.843770 20397 hybrid_clock.cc:648] HybridClock initialized: now 1790456394843747 us; error 41 us; skew 500 ppm
I20260926 20:59:54.845244 20397 webserver.cc:460] Webserver started at http://127.0.98.3:43837/ using document root <none> and password file <none>
I20260926 20:59:54.845512 20397 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:54.845582 20397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:54.845742 20397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:54.846837 20397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-2/data/instance:
uuid: "8e90ccf5ccb54e7e8eec48893fb27216"
format_stamp: "Formatted at 2026-09-26 20:59:54 on dist-test-slave-fl7g"
server_key: "3e2eac469b4d598e20d54a4e14b53c71"
server_key_iv: "3af65fd3040c6af2e6ac68f35aa829f8"
server_key_version: "encryptionkey@0"
I20260926 20:59:54.847208 20397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8e90ccf5ccb54e7e8eec48893fb27216"
format_stamp: "Formatted at 2026-09-26 20:59:54 on dist-test-slave-fl7g"
server_key: "3e2eac469b4d598e20d54a4e14b53c71"
server_key_iv: "3af65fd3040c6af2e6ac68f35aa829f8"
server_key_version: "encryptionkey@0"
I20260926 20:59:54.848658 20397 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:59:54.849442 20411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:54.849613 20397 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:59:54.849716 20397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-2/wal
uuid: "8e90ccf5ccb54e7e8eec48893fb27216"
format_stamp: "Formatted at 2026-09-26 20:59:54 on dist-test-slave-fl7g"
server_key: "3e2eac469b4d598e20d54a4e14b53c71"
server_key_iv: "3af65fd3040c6af2e6ac68f35aa829f8"
server_key_version: "encryptionkey@0"
I20260926 20:59:54.849814 20397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:54.867041 20397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:54.867394 20397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:54.867558 20397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:54.867806 20397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:59:54.868157 20397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:59:54.868212 20397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:54.868253 20397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:59:54.868281 20397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:54.873958 20397 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41783
I20260926 20:59:54.874006 20524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41783 every 8 connection(s)
I20260926 20:59:54.874398 20397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:59:54.880244 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.875251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57963 (local address 127.0.98.62:38971)
0926 20:59:54.875364 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:54.875367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:54.875587 (+ 220us) server_negotiation.cc:407] Connection header received
0926 20:59:54.876196 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:54.876198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:54.876210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:54.876264 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:54.877070 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.877583 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.878612 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.878839 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.878923 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.879920 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:54.879921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:54.879922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:54.879928 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:54.879952 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:54.879957 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:54.880012 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:54.880128 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 20:59:54.880158 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:59:54.880513 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.875212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38971 (local address 127.0.98.3:57963)
0926 20:59:54.875457 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:54.875493 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:54.876077 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:54.876365 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:54.876369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:54.876378 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:59:54.876959 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.876964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.877704 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.877705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.878516 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.878518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.878934 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.878935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.879081 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.879716 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:59:54.879716 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:59:54.879749 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:59:54.879776 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:59:54.880026 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:59:54.880033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:59:54.880036 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:59:54.880131 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 20:59:54.880245 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":93,"threads_started":1}
I20260926 20:59:54.881042 20525 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38971
I20260926 20:59:54.881167 20525 heartbeater.cc:464] Registering TS with master...
I20260926 20:59:54.881374 20525 heartbeater.cc:511] Master 127.0.98.62:38971 requested a full tablet report, sending...
I20260926 20:59:54.881783 20075 ts_manager.cc:194] Registered new tserver with Master: 8e90ccf5ccb54e7e8eec48893fb27216 (127.0.98.3:41783)
I20260926 20:59:54.882131 20075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57963
I20260926 20:59:54.884648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 20397
I20260926 20:59:54.884815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:59:54.885016 392 external_mini_cluster.cc:1428] Setting key 1404866cb16773a40aff60643e9f165b
I20260926 20:59:54.885924 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:59:54.890185 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.887333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49532 (local address 127.0.98.62:38971)
0926 20:59:54.887405 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:54.887407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:54.887564 (+ 157us) server_negotiation.cc:407] Connection header received
0926 20:59:54.887587 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:54.887587 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:54.887596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:54.887653 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:54.887798 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.888288 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.889148 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.889355 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.889412 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.889999 (+ 587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:54.890000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:54.890000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:54.890005 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:54.890018 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:54.890021 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:54.890068 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:54.890104 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 20:59:54.890123 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:59:54.891902 20075 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:59:54.893546 20075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:59:54.900178 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.896245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49730 (local address 127.0.98.1:42893)
0926 20:59:54.896483 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:54.896487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:54.896500 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:59:54.896548 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:54.896551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:54.896569 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:54.896675 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:54.896904 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.897747 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.899000 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.899748 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.899817 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.899911 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:54.900042 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 20:59:54.900080 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":92,"threads_started":1}
I20260926 20:59:54.900179 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.895967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42893 (local address 127.0.0.1:49730)
0926 20:59:54.896106 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:54.896150 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:54.896225 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:54.896740 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:54.896744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:54.896747 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:54.896816 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.896817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.897847 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.897848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.898902 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.898904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.899835 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.899836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.899916 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.899931 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 20:59:54.900021 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:59:54.900080 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":101,"threads_started":1}
I20260926 20:59:54.900707 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.895967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41783 (local address 127.0.0.1:46856)
0926 20:59:54.896523 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:54.896534 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:54.896587 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:54.897033 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:54.897033 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:54.897035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:54.897096 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.897098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.898445 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.898446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.899353 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.899356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.900386 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.900387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.900505 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.900509 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:59:54.900569 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 20:59:54.900623 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":25,"thread_start_us":103,"threads_started":1}
I20260926 20:59:54.900803 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.895971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35723 (local address 127.0.0.1:49206)
0926 20:59:54.896677 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:54.896688 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:54.896734 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:54.897020 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:54.897022 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:54.897024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:54.897112 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.897113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.898425 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.898426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.899531 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:54.899534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.900536 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.900537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:54.900591 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.900594 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:59:54.900662 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:59:54.900695 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":49,"thread_start_us":82,"threads_started":1}
I20260926 20:59:54.900802 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.896178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49206 (local address 127.0.98.2:35723)
0926 20:59:54.896459 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:54.896464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:54.896800 (+ 336us) server_negotiation.cc:407] Connection header received
0926 20:59:54.896852 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:54.896854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:54.896870 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:54.896947 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:54.897202 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.898327 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.899633 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.900392 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.900515 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.900599 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:54.900674 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 20:59:54.900720 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":104,"threads_started":1}
I20260926 20:59:54.900978 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:54.896245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.0.98.3:41783)
0926 20:59:54.896818 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:54.896822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:54.896832 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:59:54.896871 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:54.896873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:54.896887 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:54.896964 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:54.897188 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.898348 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.899498 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:54.900175 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:54.900247 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:54.900338 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:54.900823 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 20:59:54.900876 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":96,"threads_started":1}
I20260926 20:59:54.901400 20197 tablet_service.cc:1467] Processing CreateTablet for tablet f4edfd59a6934515b53e9ff622f807d0 (DEFAULT_TABLE table=test-table [id=2a8654ac60ff481e8b1dc744d29c88f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:54.901685 20197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4edfd59a6934515b53e9ff622f807d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:54.904210 20539 tablet_bootstrap.cc:492] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06: Bootstrap starting.
I20260926 20:59:54.904546 20328 tablet_service.cc:1467] Processing CreateTablet for tablet f4edfd59a6934515b53e9ff622f807d0 (DEFAULT_TABLE table=test-table [id=2a8654ac60ff481e8b1dc744d29c88f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:54.904688 20459 tablet_service.cc:1467] Processing CreateTablet for tablet f4edfd59a6934515b53e9ff622f807d0 (DEFAULT_TABLE table=test-table [id=2a8654ac60ff481e8b1dc744d29c88f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:59:54.904888 20328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4edfd59a6934515b53e9ff622f807d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:54.904983 20459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4edfd59a6934515b53e9ff622f807d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:54.905027 20539 tablet_bootstrap.cc:654] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:54.905298 20539 log.cc:824] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:54.906028 20539 tablet_bootstrap.cc:492] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06: No bootstrap required, opened a new log
I20260926 20:59:54.906109 20539 ts_tablet_manager.cc:1397] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:59:54.908185 20541 tablet_bootstrap.cc:492] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1: Bootstrap starting.
I20260926 20:59:54.908391 20539 raft_consensus.cc:348] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e90ccf5ccb54e7e8eec48893fb27216" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "95958c41aecf40d7a4ca638a39f2aea1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35723 } } peers { permanent_uuid: "87ecf16c37c24a6b8570d48b720e6f06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } }
I20260926 20:59:54.908617 20539 raft_consensus.cc:374] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:54.908663 20539 raft_consensus.cc:729] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ecf16c37c24a6b8570d48b720e6f06, State: Initialized, Role: FOLLOWER
I20260926 20:59:54.908838 20542 tablet_bootstrap.cc:492] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216: Bootstrap starting.
I20260926 20:59:54.908807 20539 consensus_queue.cc:260] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e90ccf5ccb54e7e8eec48893fb27216" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "95958c41aecf40d7a4ca638a39f2aea1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35723 } } peers { permanent_uuid: "87ecf16c37c24a6b8570d48b720e6f06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } }
I20260926 20:59:54.908926 20541 tablet_bootstrap.cc:654] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:54.909089 20539 ts_tablet_manager.cc:1428] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 20:59:54.909221 20263 heartbeater.cc:503] Master 127.0.98.62:38971 was elected leader, sending a full tablet report...
I20260926 20:59:54.909255 20541 log.cc:824] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:54.909545 20542 tablet_bootstrap.cc:654] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:54.909898 20542 log.cc:824] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:54.910144 20541 tablet_bootstrap.cc:492] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1: No bootstrap required, opened a new log
I20260926 20:59:54.910220 20541 ts_tablet_manager.cc:1397] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:59:54.910708 20542 tablet_bootstrap.cc:492] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216: No bootstrap required, opened a new log
I20260926 20:59:54.910796 20542 ts_tablet_manager.cc:1397] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:59:54.912325 20541 raft_consensus.cc:348] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e90ccf5ccb54e7e8eec48893fb27216" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "95958c41aecf40d7a4ca638a39f2aea1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35723 } } peers { permanent_uuid: "87ecf16c37c24a6b8570d48b720e6f06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } }
I20260926 20:59:54.912523 20541 raft_consensus.cc:374] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:54.912575 20541 raft_consensus.cc:729] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95958c41aecf40d7a4ca638a39f2aea1, State: Initialized, Role: FOLLOWER
I20260926 20:59:54.912706 20541 consensus_queue.cc:260] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e90ccf5ccb54e7e8eec48893fb27216" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "95958c41aecf40d7a4ca638a39f2aea1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35723 } } peers { permanent_uuid: "87ecf16c37c24a6b8570d48b720e6f06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } }
I20260926 20:59:54.912948 20541 ts_tablet_manager.cc:1428] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:59:54.913000 20394 heartbeater.cc:503] Master 127.0.98.62:38971 was elected leader, sending a full tablet report...
I20260926 20:59:54.913053 20542 raft_consensus.cc:348] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e90ccf5ccb54e7e8eec48893fb27216" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "95958c41aecf40d7a4ca638a39f2aea1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35723 } } peers { permanent_uuid: "87ecf16c37c24a6b8570d48b720e6f06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } }
I20260926 20:59:54.913204 20542 raft_consensus.cc:374] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:54.913245 20542 raft_consensus.cc:729] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e90ccf5ccb54e7e8eec48893fb27216, State: Initialized, Role: FOLLOWER
I20260926 20:59:54.913362 20542 consensus_queue.cc:260] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e90ccf5ccb54e7e8eec48893fb27216" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "95958c41aecf40d7a4ca638a39f2aea1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35723 } } peers { permanent_uuid: "87ecf16c37c24a6b8570d48b720e6f06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } }
I20260926 20:59:54.913673 20542 ts_tablet_manager.cc:1428] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:59:54.913776 20525 heartbeater.cc:503] Master 127.0.98.62:38971 was elected leader, sending a full tablet report...
W20260926 20:59:54.993239 20395 tablet.cc:2369] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:59:55.045379 20543 raft_consensus.cc:482] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:59:55.045565 20543 raft_consensus.cc:504] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e90ccf5ccb54e7e8eec48893fb27216" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "95958c41aecf40d7a4ca638a39f2aea1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35723 } } peers { permanent_uuid: "87ecf16c37c24a6b8570d48b720e6f06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } }
I20260926 20:59:55.045926 20543 leader_election.cc:288] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e90ccf5ccb54e7e8eec48893fb27216 (127.0.98.3:41783), 95958c41aecf40d7a4ca638a39f2aea1 (127.0.98.2:35723)
I20260926 20:59:55.048986 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:55.046239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50759 (local address 127.0.98.2:35723)
0926 20:59:55.046286 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:55.046288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:55.046549 (+ 261us) server_negotiation.cc:407] Connection header received
0926 20:59:55.046599 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:55.046600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:55.046613 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:55.046667 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:55.046820 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:55.047303 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:55.048249 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:55.048614 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:55.048673 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:55.048683 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:55.048868 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 20:59:55.048917 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 20:59:55.048992 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:55.046132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35723 (local address 127.0.98.1:50759)
0926 20:59:55.046462 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:55.046475 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:55.046530 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:55.046684 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:55.046685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:55.046687 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:55.046750 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:55.046752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:55.047383 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:55.047384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:55.048127 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:55.048129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:55.048707 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:55.048708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:55.048790 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:55.048794 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:59:55.048845 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 20:59:55.048877 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":154,"thread_start_us":121,"threads_started":1}
I20260926 20:59:55.049067 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:55.046217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44475 (local address 127.0.98.3:41783)
0926 20:59:55.046265 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:55.046267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:55.046285 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:59:55.046314 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:55.046315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:55.046323 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:55.046409 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:59:55.046610 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:55.047200 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:55.048198 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:55.048694 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:55.048764 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:55.048775 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:55.048939 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 20:59:55.048989 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:59:55.049273 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:55.046132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41783 (local address 127.0.98.1:44475)
0926 20:59:55.046180 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:55.046192 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:59:55.046236 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:59:55.046416 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:59:55.046417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:59:55.046420 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:59:55.046498 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:55.046500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:55.047290 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:55.047291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:55.048101 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:59:55.048104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:59:55.048782 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:59:55.048783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:59:55.048861 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:55.048864 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:59:55.048922 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:59:55.048962 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 20:59:55.049409 20348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4edfd59a6934515b53e9ff622f807d0" candidate_uuid: "87ecf16c37c24a6b8570d48b720e6f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95958c41aecf40d7a4ca638a39f2aea1" is_pre_election: true
I20260926 20:59:55.049588 20348 raft_consensus.cc:2454] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ecf16c37c24a6b8570d48b720e6f06 in term 0.
I20260926 20:59:55.049638 20479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4edfd59a6934515b53e9ff622f807d0" candidate_uuid: "87ecf16c37c24a6b8570d48b720e6f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e90ccf5ccb54e7e8eec48893fb27216" is_pre_election: true
I20260926 20:59:55.049817 20479 raft_consensus.cc:2454] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ecf16c37c24a6b8570d48b720e6f06 in term 0.
I20260926 20:59:55.049820 20151 leader_election.cc:302] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ecf16c37c24a6b8570d48b720e6f06, 95958c41aecf40d7a4ca638a39f2aea1; no voters:
I20260926 20:59:55.049968 20543 raft_consensus.cc:2781] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:59:55.050042 20543 raft_consensus.cc:482] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:59:55.050078 20543 raft_consensus.cc:3037] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:55.050798 20543 raft_consensus.cc:504] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e90ccf5ccb54e7e8eec48893fb27216" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "95958c41aecf40d7a4ca638a39f2aea1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35723 } } peers { permanent_uuid: "87ecf16c37c24a6b8570d48b720e6f06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } }
I20260926 20:59:55.050957 20543 leader_election.cc:288] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [CANDIDATE]: Term 1 election: Requested vote from peers 8e90ccf5ccb54e7e8eec48893fb27216 (127.0.98.3:41783), 95958c41aecf40d7a4ca638a39f2aea1 (127.0.98.2:35723)
I20260926 20:59:55.051131 20348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4edfd59a6934515b53e9ff622f807d0" candidate_uuid: "87ecf16c37c24a6b8570d48b720e6f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95958c41aecf40d7a4ca638a39f2aea1"
I20260926 20:59:55.051115 20479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4edfd59a6934515b53e9ff622f807d0" candidate_uuid: "87ecf16c37c24a6b8570d48b720e6f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e90ccf5ccb54e7e8eec48893fb27216"
I20260926 20:59:55.051206 20348 raft_consensus.cc:3037] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:55.051206 20479 raft_consensus.cc:3037] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:55.052126 20348 raft_consensus.cc:2454] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ecf16c37c24a6b8570d48b720e6f06 in term 1.
I20260926 20:59:55.052116 20479 raft_consensus.cc:2454] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ecf16c37c24a6b8570d48b720e6f06 in term 1.
I20260926 20:59:55.052290 20152 leader_election.cc:302] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ecf16c37c24a6b8570d48b720e6f06, 8e90ccf5ccb54e7e8eec48893fb27216; no voters:
I20260926 20:59:55.052402 20543 raft_consensus.cc:2781] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:59:55.052580 20543 raft_consensus.cc:686] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [term 1 LEADER]: Becoming Leader. State: Replica: 87ecf16c37c24a6b8570d48b720e6f06, State: Running, Role: LEADER
I20260926 20:59:55.052708 20543 consensus_queue.cc:237] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e90ccf5ccb54e7e8eec48893fb27216" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "95958c41aecf40d7a4ca638a39f2aea1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35723 } } peers { permanent_uuid: "87ecf16c37c24a6b8570d48b720e6f06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } }
I20260926 20:59:55.053608 20074 catalog_manager.cc:5495] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ecf16c37c24a6b8570d48b720e6f06 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "87ecf16c37c24a6b8570d48b720e6f06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e90ccf5ccb54e7e8eec48893fb27216" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95958c41aecf40d7a4ca638a39f2aea1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ecf16c37c24a6b8570d48b720e6f06" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } health_report { overall_health: HEALTHY } } }
I20260926 20:59:55.082228 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:55.079632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49744 (local address 127.0.98.1:42893)
0926 20:59:55.079691 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:55.079693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:55.079702 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:59:55.079746 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:55.079747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:55.079767 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:55.079862 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:59:55.080058 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:55.080540 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:55.081336 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:55.081546 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:55.081621 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:55.081826 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:59:55.081934 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:59:55.082011 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:55.082116 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 20:59:55.082141 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:59:55.084308 20479 raft_consensus.cc:1261] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216 [term 1 FOLLOWER]: Refusing update from remote peer 87ecf16c37c24a6b8570d48b720e6f06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:59:55.084326 20348 raft_consensus.cc:1261] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1 [term 1 FOLLOWER]: Refusing update from remote peer 87ecf16c37c24a6b8570d48b720e6f06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:59:55.084795 20549 consensus_queue.cc:1035] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e90ccf5ccb54e7e8eec48893fb27216" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:59:55.084995 20543 consensus_queue.cc:1035] T f4edfd59a6934515b53e9ff622f807d0 P 87ecf16c37c24a6b8570d48b720e6f06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95958c41aecf40d7a4ca638a39f2aea1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:59:55.088340 20553 mvcc.cc:205] Tried to move back new op lower bound from 7333709394261868544 to 7333709394136563712. Current Snapshot: MvccSnapshot[applied={T|T < 7333709394261868544}]
I20260926 20:59:55.089908 20074 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:59:55.090010 20074 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:59:55.090781 20074 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d5a620f462a4d4a93f8d76374912e4d: Sending DeleteTablet for 3 replicas of tablet f4edfd59a6934515b53e9ff622f807d0
I20260926 20:59:55.091136 20328 tablet_service.cc:1514] Processing DeleteTablet for tablet f4edfd59a6934515b53e9ff622f807d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:55 UTC) from {username='slave'} at 127.0.0.1:49206
I20260926 20:59:55.091152 20459 tablet_service.cc:1514] Processing DeleteTablet for tablet f4edfd59a6934515b53e9ff622f807d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:55 UTC) from {username='slave'} at 127.0.0.1:46856
I20260926 20:59:55.091377 20561 tablet_replica.cc:330] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1: stopping tablet replica
I20260926 20:59:55.091473 20561 raft_consensus.cc:2229] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:59:55.091578 20561 raft_consensus.cc:2258] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:59:55.091708 20197 tablet_service.cc:1514] Processing DeleteTablet for tablet f4edfd59a6934515b53e9ff622f807d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:59:55 UTC) from {username='slave'} at 127.0.0.1:49730
I20260926 20:59:55.091758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 20135
I20260926 20:59:55.092031 20561 ts_tablet_manager.cc:1905] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:59:55.092461 20562 tablet_replica.cc:330] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216: stopping tablet replica
I20260926 20:59:55.092597 20562 raft_consensus.cc:2229] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:59:55.092757 20562 raft_consensus.cc:2258] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:59:55.093313 20562 ts_tablet_manager.cc:1905] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:59:55.094313 20561 ts_tablet_manager.cc:1918] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:59:55.094393 20561 log.cc:1196] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-1/wal/wals/f4edfd59a6934515b53e9ff622f807d0
I20260926 20:59:55.094655 20561 ts_tablet_manager.cc:1939] T f4edfd59a6934515b53e9ff622f807d0 P 95958c41aecf40d7a4ca638a39f2aea1: Deleting consensus metadata
I20260926 20:59:55.095290 20060 catalog_manager.cc:4841] TS 95958c41aecf40d7a4ca638a39f2aea1 (127.0.98.2:35723): tablet f4edfd59a6934515b53e9ff622f807d0 (table test-table [id=2a8654ac60ff481e8b1dc744d29c88f8]) successfully deleted
I20260926 20:59:55.095427 20562 ts_tablet_manager.cc:1918] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:59:55.095498 20562 log.cc:1196] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216: Deleting WAL directory at /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790456150796563-392-0/minicluster-data/ts-2/wal/wals/f4edfd59a6934515b53e9ff622f807d0
I20260926 20:59:55.095737 20562 ts_tablet_manager.cc:1939] T f4edfd59a6934515b53e9ff622f807d0 P 8e90ccf5ccb54e7e8eec48893fb27216: Deleting consensus metadata
I20260926 20:59:55.096431 20061 catalog_manager.cc:4841] TS 8e90ccf5ccb54e7e8eec48893fb27216 (127.0.98.3:41783): tablet f4edfd59a6934515b53e9ff622f807d0 (table test-table [id=2a8654ac60ff481e8b1dc744d29c88f8]) successfully deleted
W20260926 20:59:55.098275 20059 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42893 (error 108)
I20260926 20:59:55.098491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 20266
I20260926 20:59:55.099078 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:55.098613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42893 (local address 127.0.0.1:49754)
0926 20:59:55.099002 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0926 20:59:55.099051 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366}
W20260926 20:59:55.099352 20059 catalog_manager.cc:4584] TS 87ecf16c37c24a6b8570d48b720e6f06 (127.0.98.1:42893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f4edfd59a6934515b53e9ff622f807d0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42893: connect: Connection refused (error 111)
I20260926 20:59:55.103746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 20397
I20260926 20:59:55.108187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 20044
2026-09-26T20:59:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (802 ms)
[----------] 6 tests from EncryptionPolicyTest (8925 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:59:55 dist-test-slave-fl7g krb5kdc[20566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:59:55 dist-test-slave-fl7g krb5kdc[20566](info): set up 2 sockets
Sep 26 20:59:55 dist-test-slave-fl7g krb5kdc[20566](info): commencing operation
krb5kdc: starting...
W20260926 20:59:56.160516 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:59:56 dist-test-slave-fl7g krb5kdc[20566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:59:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:59:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:59:56.288154 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45207
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:56.372136 20579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:56.372375 20579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:56.372400 20579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:56.373668 20579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:59:56.373714 20579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:56.373729 20579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:56.373742 20579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:59:56.373754 20579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:59:56.375356 20579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45207
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:56.375653 20579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:56.375905 20579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:56.381348 20585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:56.381341 20587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:56.381404 20579 server_base.cc:1034] running on GCE node
W20260926 20:59:56.381361 20584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:56.381923 20579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:56.382288 20579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:56.383479 20579 hybrid_clock.cc:648] HybridClock initialized: now 1790456396383466 us; error 33 us; skew 500 ppm
Sep 26 20:59:56 dist-test-slave-fl7g krb5kdc[20566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:56.392503 20579 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:59:56.393085 20579 webserver.cc:460] Webserver started at http://127.0.0.1:33385/ using document root <none> and password file <none>
I20260926 20:59:56.393370 20579 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:56.393419 20579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:56.393580 20579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:56.394690 20579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "f5b5136dc706482c88b0f28e0820a15b"
format_stamp: "Formatted at 2026-09-26 20:59:56 on dist-test-slave-fl7g"
server_key: "ebb057ddd8fb8510a41334c887cfb6ee"
server_key_iv: "84dc86568e436bd219dbfc08e38d5393"
server_key_version: "encryptionkey@0"
I20260926 20:59:56.395066 20579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "f5b5136dc706482c88b0f28e0820a15b"
format_stamp: "Formatted at 2026-09-26 20:59:56 on dist-test-slave-fl7g"
server_key: "ebb057ddd8fb8510a41334c887cfb6ee"
server_key_iv: "84dc86568e436bd219dbfc08e38d5393"
server_key_version: "encryptionkey@0"
I20260926 20:59:56.396590 20579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:59:56.397535 20594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:56.397737 20579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:56.397840 20579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "f5b5136dc706482c88b0f28e0820a15b"
format_stamp: "Formatted at 2026-09-26 20:59:56 on dist-test-slave-fl7g"
server_key: "ebb057ddd8fb8510a41334c887cfb6ee"
server_key_iv: "84dc86568e436bd219dbfc08e38d5393"
server_key_version: "encryptionkey@0"
I20260926 20:59:56.397902 20579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:56.410312 20579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:56.412882 20579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:56.413007 20579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:56.417014 20579 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45207
I20260926 20:59:56.417078 20646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45207 every 8 connection(s)
I20260926 20:59:56.417430 20579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:59:56.418439 20647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:56.420971 20647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b: Bootstrap starting.
I20260926 20:59:56.421592 20647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:56.421903 20647 log.cc:824] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:56.422667 20647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b: No bootstrap required, opened a new log
I20260926 20:59:56.423456 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 20579
I20260926 20:59:56.423552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:59:56.423772 392 external_mini_cluster.cc:1428] Setting key c19a7df7f2d1af3a8e391ee2ade59cc4
I20260926 20:59:56.424858 20647 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b5136dc706482c88b0f28e0820a15b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45207 } }
I20260926 20:59:56.425051 20647 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:56.425094 20647 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b5136dc706482c88b0f28e0820a15b, State: Initialized, Role: FOLLOWER
I20260926 20:59:56.425238 20647 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b5136dc706482c88b0f28e0820a15b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45207 } }
I20260926 20:59:56.425331 20647 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:59:56.425367 20647 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:59:56.425420 20647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:56.426472 20647 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b5136dc706482c88b0f28e0820a15b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45207 } }
I20260926 20:59:56.426565 20647 leader_election.cc:302] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5b5136dc706482c88b0f28e0820a15b; no voters:
I20260926 20:59:56.426780 20647 leader_election.cc:288] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:59:56.426805 20652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:59:56.426911 20652 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b [term 1 LEADER]: Becoming Leader. State: Replica: f5b5136dc706482c88b0f28e0820a15b, State: Running, Role: LEADER
I20260926 20:59:56.426991 20652 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b5136dc706482c88b0f28e0820a15b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45207 } }
I20260926 20:59:56.427052 20647 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:59:56.427364 20652 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5b5136dc706482c88b0f28e0820a15b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b5136dc706482c88b0f28e0820a15b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45207 } } }
I20260926 20:59:56.427491 20652 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:56.427647 20652 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5b5136dc706482c88b0f28e0820a15b. Latest consensus state: current_term: 1 leader_uuid: "f5b5136dc706482c88b0f28e0820a15b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b5136dc706482c88b0f28e0820a15b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45207 } } }
I20260926 20:59:56.427718 20652 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:56.427845 20659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:59:56.428388 20659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:59:56.430033 20659 catalog_manager.cc:1347] Generated new cluster ID: e1a04442b4ad46b89399e6e18da2fa94
I20260926 20:59:56.430088 20659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:59:56 dist-test-slave-fl7g krb5kdc[20566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:59:56.435014 20651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:56.424383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38035 (local address 127.0.0.1:45207)
0926 20:59:56.424606 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:56.424614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:56.424639 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:59:56.424833 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:56.424840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:56.424887 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:56.425009 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:56.425845 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:56.426861 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:56.427946 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:56.428704 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:56.428811 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:56.431481 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:56.431485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:56.431485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:56.431497 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:56.433258 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:56.433661 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:56.433664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:56.433664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:56.433782 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:56.434023 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:56.434024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:56.434024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:56.434266 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:56.434454 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:56.434539 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 20:59:56.434644 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":143,"threads_started":1}
I20260926 20:59:56.436075 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:59:56.447091 20651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:56.437533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42883 (local address 127.0.0.1:45207)
0926 20:59:56.437574 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:56.437576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:56.440641 (+ 3065us) server_negotiation.cc:407] Connection header received
0926 20:59:56.440723 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:56.440724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:56.440735 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:56.440820 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:56.441376 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:56.441928 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:56.442707 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:56.442946 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:56.443050 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:56.444276 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:56.444277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:56.444278 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:59:56.444285 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:56.445862 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:56.446252 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:56.446254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:56.446254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:56.446320 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:59:56.446541 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:59:56.446543 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:59:56.446543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:59:56.446663 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:56.446778 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:56.446900 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 20:59:56.446968 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:59:56.480913 20659 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:59:56.481482 20659 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:59:56.491422 20659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5b5136dc706482c88b0f28e0820a15b: Generated new TSK 0
I20260926 20:59:56.491657 20659 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:59:56.502571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 20579
W20260926 20:59:56.506884 20564 connection.cc:535] client connection to 127.0.0.1:45207 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45207): Connection reset by peer (error 104)
2026-09-26T20:59:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:59:56 dist-test-slave-fl7g krb5kdc[20678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:59:56 dist-test-slave-fl7g krb5kdc[20678](info): set up 2 sockets
Sep 26 20:59:56 dist-test-slave-fl7g krb5kdc[20678](info): commencing operation
krb5kdc: starting...
W20260926 20:59:57.564623 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:59:57 dist-test-slave-fl7g krb5kdc[20678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:59:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:59:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:59:57.689141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:57.771028 20691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:57.771292 20691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:57.771335 20691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:57.772526 20691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:59:57.772603 20691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:57.772627 20691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:57.772650 20691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:59:57.772678 20691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:59:57.774200 20691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43875
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:57.774489 20691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:57.774736 20691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:57.779834 20696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:57.779927 20691 server_base.cc:1034] running on GCE node
W20260926 20:59:57.779888 20697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:57.779870 20699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:57.780300 20691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:57.780659 20691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:57.781839 20691 hybrid_clock.cc:648] HybridClock initialized: now 1790456397781822 us; error 30 us; skew 500 ppm
Sep 26 20:59:57 dist-test-slave-fl7g krb5kdc[20678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:57.784446 20691 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:59:57.784871 20691 webserver.cc:460] Webserver started at http://127.0.0.1:39069/ using document root <none> and password file <none>
I20260926 20:59:57.785063 20691 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:57.785106 20691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:57.785269 20691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:57.786407 20691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "543ff1bacccf4d60aa405b46b2b2a5bc"
format_stamp: "Formatted at 2026-09-26 20:59:57 on dist-test-slave-fl7g"
server_key: "2f51d3092a1a5afc332f785307084874"
server_key_iv: "67cc75c1b4458468d0df3ae95f139d4e"
server_key_version: "encryptionkey@0"
I20260926 20:59:57.786717 20691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "543ff1bacccf4d60aa405b46b2b2a5bc"
format_stamp: "Formatted at 2026-09-26 20:59:57 on dist-test-slave-fl7g"
server_key: "2f51d3092a1a5afc332f785307084874"
server_key_iv: "67cc75c1b4458468d0df3ae95f139d4e"
server_key_version: "encryptionkey@0"
I20260926 20:59:57.788121 20691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:59:57.789039 20706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:57.789204 20691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:59:57.789295 20691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "543ff1bacccf4d60aa405b46b2b2a5bc"
format_stamp: "Formatted at 2026-09-26 20:59:57 on dist-test-slave-fl7g"
server_key: "2f51d3092a1a5afc332f785307084874"
server_key_iv: "67cc75c1b4458468d0df3ae95f139d4e"
server_key_version: "encryptionkey@0"
I20260926 20:59:57.789361 20691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:57.802459 20691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:57.805029 20691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:57.805159 20691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:57.809491 20691 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43875
I20260926 20:59:57.809559 20758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43875 every 8 connection(s)
I20260926 20:59:57.809931 20691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:59:57.810856 20759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:57.813311 20759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc: Bootstrap starting.
I20260926 20:59:57.813935 20759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:57.814244 20759 log.cc:824] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:57.814261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 20691
I20260926 20:59:57.814376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:59:57.814687 392 external_mini_cluster.cc:1428] Setting key 057bf923003070d6190552792d22625e
I20260926 20:59:57.815147 20759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc: No bootstrap required, opened a new log
I20260926 20:59:57.817145 20759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543ff1bacccf4d60aa405b46b2b2a5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43875 } }
I20260926 20:59:57.817317 20759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:57.817380 20759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 543ff1bacccf4d60aa405b46b2b2a5bc, State: Initialized, Role: FOLLOWER
I20260926 20:59:57.817524 20759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543ff1bacccf4d60aa405b46b2b2a5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43875 } }
I20260926 20:59:57.817621 20759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:59:57.817665 20759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:59:57.817710 20759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:57.818311 20759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543ff1bacccf4d60aa405b46b2b2a5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43875 } }
I20260926 20:59:57.818401 20759 leader_election.cc:302] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 543ff1bacccf4d60aa405b46b2b2a5bc; no voters:
I20260926 20:59:57.818578 20759 leader_election.cc:288] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:59:57.818603 20764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:59:57.818727 20764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc [term 1 LEADER]: Becoming Leader. State: Replica: 543ff1bacccf4d60aa405b46b2b2a5bc, State: Running, Role: LEADER
I20260926 20:59:57.818859 20759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:59:57.818862 20764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543ff1bacccf4d60aa405b46b2b2a5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43875 } }
I20260926 20:59:57.819298 20765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "543ff1bacccf4d60aa405b46b2b2a5bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543ff1bacccf4d60aa405b46b2b2a5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43875 } } }
I20260926 20:59:57.819437 20765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:57.819308 20766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 543ff1bacccf4d60aa405b46b2b2a5bc. Latest consensus state: current_term: 1 leader_uuid: "543ff1bacccf4d60aa405b46b2b2a5bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543ff1bacccf4d60aa405b46b2b2a5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43875 } } }
I20260926 20:59:57.819587 20766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc [sys.catalog]: This master's current role is: LEADER
W20260926 20:59:57.820996 20780 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:59:57.821055 20780 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:59:57.821112 20772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:59:57.821314 20772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:59:57.822696 20763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:57.815373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51319 (local address 127.0.0.1:43875)
0926 20:59:57.815550 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:57.815560 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:57.816666 (+ 1106us) server_negotiation.cc:407] Connection header received
0926 20:59:57.817154 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:57.817163 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:57.817224 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:57.817416 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:57.817672 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:57.819260 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:57.820391 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:57.821114 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:57.821220 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:57.821994 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:57.821998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:57.821999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:57.822214 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:57.822258 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:57.822276 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:57.822364 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:57.822444 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 20:59:57.822473 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260926 20:59:57.822847 20772 catalog_manager.cc:1347] Generated new cluster ID: f81c4ed452614f57acb0aa6947d19cd4
I20260926 20:59:57.823086 20772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:59:57.824057 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:59:57.832497 20763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:57.828784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54543 (local address 127.0.0.1:43875)
0926 20:59:57.828824 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:57.828826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:57.828954 (+ 128us) server_negotiation.cc:407] Connection header received
0926 20:59:57.829017 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:57.829018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:57.829037 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:57.829100 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:57.829288 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:57.830014 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:57.831102 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:59:57.831409 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:59:57.831472 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:59:57.832218 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:57.832219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:57.832219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:57.832223 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:57.832238 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:57.832242 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:57.832296 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:57.832376 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 20:59:57.832388 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:59:57.834201 20772 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:59:57.834697 20772 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:59:57.838737 20772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 543ff1bacccf4d60aa405b46b2b2a5bc: Generated new TSK 0
I20260926 20:59:57.838948 20772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:59:57.845391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 20691
W20260926 20:59:57.849819 20676 connection.cc:535] client connection to 127.0.0.1:43875 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43875): Connection reset by peer (error 104)
2026-09-26T20:59:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:59:57 dist-test-slave-fl7g krb5kdc[20790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:59:57 dist-test-slave-fl7g krb5kdc[20790](info): set up 2 sockets
Sep 26 20:59:57 dist-test-slave-fl7g krb5kdc[20790](info): commencing operation
krb5kdc: starting...
W20260926 20:59:58.880645 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 20:59:58 dist-test-slave-fl7g krb5kdc[20790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:59:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:59:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:59:59.008055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:59:59.091360 20803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:59:59.091661 20803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:59:59.091709 20803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:59:59.093050 20803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:59:59.093146 20803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:59:59.093173 20803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:59:59.093195 20803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:59:59.093223 20803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:59:59.094918 20803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42515
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:59:59.095234 20803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:59:59.095485 20803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:59:59.100605 20808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:59.100658 20809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:59:59.100605 20811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:59:59.100831 20803 server_base.cc:1034] running on GCE node
I20260926 20:59:59.101154 20803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:59:59.101509 20803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:59:59.102705 20803 hybrid_clock.cc:648] HybridClock initialized: now 1790456399102700 us; error 46 us; skew 500 ppm
Sep 26 20:59:59 dist-test-slave-fl7g krb5kdc[20790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:59:59.105587 20803 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:59:59.106074 20803 webserver.cc:460] Webserver started at http://127.0.0.1:45517/ using document root <none> and password file <none>
I20260926 20:59:59.106288 20803 fs_manager.cc:359] Metadata directory not provided
I20260926 20:59:59.106329 20803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:59:59.106420 20803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:59:59.107481 20803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "d2e6eb3029e343e9b92caca017f57a0c"
format_stamp: "Formatted at 2026-09-26 20:59:59 on dist-test-slave-fl7g"
server_key: "dc2c9871928bce16d7bef817fc359dbb"
server_key_iv: "d6c76492f70c4588f112c84bf815c11b"
server_key_version: "encryptionkey@0"
I20260926 20:59:59.107795 20803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "d2e6eb3029e343e9b92caca017f57a0c"
format_stamp: "Formatted at 2026-09-26 20:59:59 on dist-test-slave-fl7g"
server_key: "dc2c9871928bce16d7bef817fc359dbb"
server_key_iv: "d6c76492f70c4588f112c84bf815c11b"
server_key_version: "encryptionkey@0"
I20260926 20:59:59.109295 20803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:59:59.110175 20818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:59:59.110383 20803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:59:59.110481 20803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "d2e6eb3029e343e9b92caca017f57a0c"
format_stamp: "Formatted at 2026-09-26 20:59:59 on dist-test-slave-fl7g"
server_key: "dc2c9871928bce16d7bef817fc359dbb"
server_key_iv: "d6c76492f70c4588f112c84bf815c11b"
server_key_version: "encryptionkey@0"
I20260926 20:59:59.110551 20803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:59:59.114040 20803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:59:59.116576 20803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:59:59.116700 20803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:59:59.120622 20803 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42515
I20260926 20:59:59.120700 20870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42515 every 8 connection(s)
I20260926 20:59:59.121047 20803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:59:59.122121 20871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:59:59.123003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 20803
I20260926 20:59:59.123134 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 20:59:59.123346 392 external_mini_cluster.cc:1428] Setting key f606b25bb8a1e43cfd94d23dd61fb791
I20260926 20:59:59.124987 20871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c: Bootstrap starting.
I20260926 20:59:59.125739 20871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c: Neither blocks nor log segments found. Creating new log.
I20260926 20:59:59.126045 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:59.123945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45743 (local address 127.0.0.1:42515)
0926 20:59:59.124122 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:59.124130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:59.124156 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:59:59.124775 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:59.124783 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:59.124812 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:59.124941 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:59.125627 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:59.125630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:59.125630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:59.125634 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:59.125671 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:59.125681 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:59.125758 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:59.125857 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 20:59:59.125905 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260926 20:59:59.126078 20871 log.cc:824] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:59:59.126860 20871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c: No bootstrap required, opened a new log
I20260926 20:59:59.127169 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:59:59.129101 20871 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e6eb3029e343e9b92caca017f57a0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42515 } }
I20260926 20:59:59.129294 20871 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:59:59.129345 20871 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e6eb3029e343e9b92caca017f57a0c, State: Initialized, Role: FOLLOWER
I20260926 20:59:59.129474 20871 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e6eb3029e343e9b92caca017f57a0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42515 } }
I20260926 20:59:59.129587 20871 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:59:59.129635 20871 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:59:59.129686 20871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:59:59.129824 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:59:59.128614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44959 (local address 127.0.0.1:42515)
0926 20:59:59.128647 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 20:59:59.128649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:59:59.128786 (+ 137us) server_negotiation.cc:407] Connection header received
0926 20:59:59.128860 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:59:59.128860 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:59:59.128865 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:59:59.128909 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:59:59.129572 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:59:59.129574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:59:59.129574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:59:59.129579 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:59:59.129593 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:59:59.129597 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:59:59.129645 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:59:59.129747 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 20:59:59.129760 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 20:59:59.130453 20871 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e6eb3029e343e9b92caca017f57a0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42515 } }
I20260926 20:59:59.130569 20871 leader_election.cc:302] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2e6eb3029e343e9b92caca017f57a0c; no voters:
I20260926 20:59:59.130762 20871 leader_election.cc:288] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:59:59.130851 20881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:59:59.131013 20881 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c [term 1 LEADER]: Becoming Leader. State: Replica: d2e6eb3029e343e9b92caca017f57a0c, State: Running, Role: LEADER
I20260926 20:59:59.131033 20871 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:59:59.131105 20881 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e6eb3029e343e9b92caca017f57a0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42515 } }
I20260926 20:59:59.131515 20882 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2e6eb3029e343e9b92caca017f57a0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e6eb3029e343e9b92caca017f57a0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42515 } } }
I20260926 20:59:59.131671 20882 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c [sys.catalog]: This master's current role is: LEADER
I20260926 20:59:59.131515 20883 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2e6eb3029e343e9b92caca017f57a0c. Latest consensus state: current_term: 1 leader_uuid: "d2e6eb3029e343e9b92caca017f57a0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e6eb3029e343e9b92caca017f57a0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42515 } } }
I20260926 20:59:59.131918 20883 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c [sys.catalog]: This master's current role is: LEADER
W20260926 20:59:59.132907 20897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:59:59.132998 20897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:59:59.133049 20890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:59:59.133215 20890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:59:59.134833 20890 catalog_manager.cc:1347] Generated new cluster ID: 07a297376b95411dbf8eef757c30c686
I20260926 20:59:59.134892 20890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:59:59.163259 20890 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:59:59.163488 20890 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:59:59.172536 20890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2e6eb3029e343e9b92caca017f57a0c: Generated new TSK 0
I20260926 20:59:59.172791 20890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:59:59.183398 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 20803
2026-09-26T20:59:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:59:59 dist-test-slave-fl7g krb5kdc[20902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:59:59 dist-test-slave-fl7g krb5kdc[20902](info): set up 2 sockets
Sep 26 20:59:59 dist-test-slave-fl7g krb5kdc[20902](info): commencing operation
krb5kdc: starting...
W20260926 21:00:00.212453 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 21:00:00 dist-test-slave-fl7g krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:00:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:00:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:00:00.336735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43431
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:00:00.420789 20915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:00.421017 20915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:00.421041 20915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:00.422330 20915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:00.422452 20915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:00.422492 20915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:00.422520 20915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:00.422534 20915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:00.424124 20915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:00.424381 20915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:00.424700 20915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:00.430001 20923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:00.430032 20915 server_base.cc:1034] running on GCE node
W20260926 21:00:00.430001 20920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:00.430033 20921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:00.430483 20915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:00.430817 20915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:00.431995 20915 hybrid_clock.cc:648] HybridClock initialized: now 1790456400431975 us; error 39 us; skew 500 ppm
Sep 26 21:00:00 dist-test-slave-fl7g krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:00.434669 20915 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:00:00.435076 20915 webserver.cc:460] Webserver started at http://127.0.0.1:39835/ using document root <none> and password file <none>
I20260926 21:00:00.435288 20915 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:00.435330 20915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:00.435420 20915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:00.436550 20915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "9c352372920049cb8744a0a85f8a41e2"
format_stamp: "Formatted at 2026-09-26 21:00:00 on dist-test-slave-fl7g"
server_key: "65990e32485150aaa04ed4d5764392a2"
server_key_iv: "4208cc31ed3c0b953a7dd372a0431dff"
server_key_version: "encryptionkey@0"
I20260926 21:00:00.436884 20915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "9c352372920049cb8744a0a85f8a41e2"
format_stamp: "Formatted at 2026-09-26 21:00:00 on dist-test-slave-fl7g"
server_key: "65990e32485150aaa04ed4d5764392a2"
server_key_iv: "4208cc31ed3c0b953a7dd372a0431dff"
server_key_version: "encryptionkey@0"
I20260926 21:00:00.438234 20915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:00:00.439088 20930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:00.439276 20915 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:00:00.439375 20915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "9c352372920049cb8744a0a85f8a41e2"
format_stamp: "Formatted at 2026-09-26 21:00:00 on dist-test-slave-fl7g"
server_key: "65990e32485150aaa04ed4d5764392a2"
server_key_iv: "4208cc31ed3c0b953a7dd372a0431dff"
server_key_version: "encryptionkey@0"
I20260926 21:00:00.439450 20915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:00.456835 20915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:00.459273 20915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:00.459403 20915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:00.463766 20915 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43431
I20260926 21:00:00.463836 20982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43431 every 8 connection(s)
I20260926 21:00:00.464193 20915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 21:00:00.465157 20983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:00.467481 20983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2: Bootstrap starting.
I20260926 21:00:00.468034 20983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:00.468281 20983 log.cc:824] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:00.469074 20983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2: No bootstrap required, opened a new log
I20260926 21:00:00.470619 20983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c352372920049cb8744a0a85f8a41e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43431 } }
I20260926 21:00:00.470798 20983 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:00:00.470822 20983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c352372920049cb8744a0a85f8a41e2, State: Initialized, Role: FOLLOWER
I20260926 21:00:00.470974 20983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c352372920049cb8744a0a85f8a41e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43431 } }
I20260926 21:00:00.471058 20983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:00:00.471082 20983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:00:00.471112 20983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:00.471608 20983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c352372920049cb8744a0a85f8a41e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43431 } }
I20260926 21:00:00.471704 20983 leader_election.cc:302] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c352372920049cb8744a0a85f8a41e2; no voters:
I20260926 21:00:00.471840 20983 leader_election.cc:288] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:00:00.471946 20986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:00:00.472087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 20915
I20260926 21:00:00.472149 20983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:00.472204 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 21:00:00.472220 20986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2 [term 1 LEADER]: Becoming Leader. State: Replica: 9c352372920049cb8744a0a85f8a41e2, State: Running, Role: LEADER
I20260926 21:00:00.472350 20986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c352372920049cb8744a0a85f8a41e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43431 } }
I20260926 21:00:00.472457 392 external_mini_cluster.cc:1428] Setting key 4fb32418627b7a808a64feff5c69b888
I20260926 21:00:00.472808 20987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c352372920049cb8744a0a85f8a41e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c352372920049cb8744a0a85f8a41e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43431 } } }
I20260926 21:00:00.472913 20987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2 [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:00.472841 20988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c352372920049cb8744a0a85f8a41e2. Latest consensus state: current_term: 1 leader_uuid: "9c352372920049cb8744a0a85f8a41e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c352372920049cb8744a0a85f8a41e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43431 } } }
I20260926 21:00:00.473207 20988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2 [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:00.473407 20991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:00:00.474035 20991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:00:00.475764 20991 catalog_manager.cc:1347] Generated new cluster ID: 37f9830d5ca541c7b7dc007fb9da1f89
I20260926 21:00:00.475824 20991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 21:00:00 dist-test-slave-fl7g krb5kdc[20902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 21:00:00.483323 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:00.473066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57479 (local address 127.0.0.1:43431)
0926 21:00:00.473387 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:00.473398 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:00.473440 (+ 42us) server_negotiation.cc:407] Connection header received
0926 21:00:00.473605 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:00.473613 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:00.473675 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:00.473854 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:00.474753 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:00.476030 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:00.476955 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:00.477390 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:00.477525 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:00.479888 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:00.479892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:00.479893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:00.479908 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:00.481691 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:00.482089 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:00.482092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:00.482092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:00.482205 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:00.482440 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:00.482441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:00.482441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:00.482705 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:00.482901 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:00.482968 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 21:00:00.483112 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":80,"threads_started":1}
I20260926 21:00:00.484280 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:00:00.493499 20991 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:00:00.494238 20991 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:00:00.495476 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:00.488808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39213 (local address 127.0.0.1:43431)
0926 21:00:00.488855 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:00.488856 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:00.488973 (+ 117us) server_negotiation.cc:407] Connection header received
0926 21:00:00.489055 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:00.489056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:00.489067 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:00.489117 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:00.489678 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:00.490214 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:00.490973 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:00.491208 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:00.491301 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:00.492663 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:00.492665 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:00.492666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:00.492674 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:00.494281 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:00.494662 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:00.494663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:00.494664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:00.494738 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:00.494939 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:00.494940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:00.494941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:00.495055 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:00.495180 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:00.495265 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 21:00:00.495362 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 21:00:00.499058 20991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c352372920049cb8744a0a85f8a41e2: Generated new TSK 0
I20260926 21:00:00.499285 20991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:00:00.504755 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 20915
2026-09-26T21:00:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:00:00 dist-test-slave-fl7g krb5kdc[21014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:00:00 dist-test-slave-fl7g krb5kdc[21014](info): set up 2 sockets
Sep 26 21:00:00 dist-test-slave-fl7g krb5kdc[21014](info): commencing operation
krb5kdc: starting...
W20260926 21:00:01.564693 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 21:00:01 dist-test-slave-fl7g krb5kdc[21014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:00:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:00:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:00:01.688953 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:00:01.771515 21027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:01.771740 21027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:01.771763 21027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:01.773038 21027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:01.773085 21027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:01.773100 21027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:01.773113 21027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:01.773175 21027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:01.774689 21027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:01.774969 21027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:01.775210 21027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 21:00:01.780339 21027 server_base.cc:1034] running on GCE node
W20260926 21:00:01.780310 21032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:01.780296 21035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:01.780310 21033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:01.780910 21027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:01.781242 21027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:01.782400 21027 hybrid_clock.cc:648] HybridClock initialized: now 1790456401782387 us; error 31 us; skew 500 ppm
Sep 26 21:00:01 dist-test-slave-fl7g krb5kdc[21014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:01.789215 21027 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:00:01.789722 21027 webserver.cc:460] Webserver started at http://127.0.0.1:39273/ using document root <none> and password file <none>
I20260926 21:00:01.789984 21027 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:01.790057 21027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:01.790206 21027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:01.791334 21027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "cc0d7490454946a3a1cfc3ef8a94691a"
format_stamp: "Formatted at 2026-09-26 21:00:01 on dist-test-slave-fl7g"
server_key: "4b7e95a79469dc1bdd75f9748a33e630"
server_key_iv: "7dab4b6c4d8c74e112e7199a2dfe43e2"
server_key_version: "encryptionkey@0"
I20260926 21:00:01.791707 21027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "cc0d7490454946a3a1cfc3ef8a94691a"
format_stamp: "Formatted at 2026-09-26 21:00:01 on dist-test-slave-fl7g"
server_key: "4b7e95a79469dc1bdd75f9748a33e630"
server_key_iv: "7dab4b6c4d8c74e112e7199a2dfe43e2"
server_key_version: "encryptionkey@0"
I20260926 21:00:01.793258 21027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 21:00:01.794198 21042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:01.794440 21027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:00:01.794566 21027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "cc0d7490454946a3a1cfc3ef8a94691a"
format_stamp: "Formatted at 2026-09-26 21:00:01 on dist-test-slave-fl7g"
server_key: "4b7e95a79469dc1bdd75f9748a33e630"
server_key_iv: "7dab4b6c4d8c74e112e7199a2dfe43e2"
server_key_version: "encryptionkey@0"
I20260926 21:00:01.794655 21027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:01.807291 21027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:01.810289 21027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:01.810477 21027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:01.814596 21027 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43419
I20260926 21:00:01.814628 21094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43419 every 8 connection(s)
I20260926 21:00:01.815008 21027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 21:00:01.816088 21095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:01.819614 21095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a: Bootstrap starting.
I20260926 21:00:01.820242 21095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:01.820600 21095 log.cc:824] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:01.821360 21095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a: No bootstrap required, opened a new log
I20260926 21:00:01.822974 21095 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0d7490454946a3a1cfc3ef8a94691a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43419 } }
I20260926 21:00:01.823145 21095 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:00:01.823194 21095 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc0d7490454946a3a1cfc3ef8a94691a, State: Initialized, Role: FOLLOWER
I20260926 21:00:01.823379 21095 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0d7490454946a3a1cfc3ef8a94691a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43419 } }
I20260926 21:00:01.823463 21095 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:00:01.823500 21095 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:00:01.823551 21095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:01.824100 21095 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0d7490454946a3a1cfc3ef8a94691a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43419 } }
I20260926 21:00:01.824247 21095 leader_election.cc:302] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc0d7490454946a3a1cfc3ef8a94691a; no voters:
I20260926 21:00:01.824438 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 21027
I20260926 21:00:01.824489 21095 leader_election.cc:288] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:00:01.824515 21098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:00:01.824582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 21:00:01.824630 21098 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a [term 1 LEADER]: Becoming Leader. State: Replica: cc0d7490454946a3a1cfc3ef8a94691a, State: Running, Role: LEADER
I20260926 21:00:01.824702 21098 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0d7490454946a3a1cfc3ef8a94691a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43419 } }
I20260926 21:00:01.824755 21095 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:01.824786 392 external_mini_cluster.cc:1428] Setting key 6154bf8dbe43f631f75fd35ea019cc1a
I20260926 21:00:01.825215 21099 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc0d7490454946a3a1cfc3ef8a94691a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0d7490454946a3a1cfc3ef8a94691a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43419 } } }
I20260926 21:00:01.825314 21099 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:01.825507 21100 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc0d7490454946a3a1cfc3ef8a94691a. Latest consensus state: current_term: 1 leader_uuid: "cc0d7490454946a3a1cfc3ef8a94691a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0d7490454946a3a1cfc3ef8a94691a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43419 } } }
I20260926 21:00:01.825573 21100 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:01.825690 21108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:00:01.826294 21108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:00:01.828274 21108 catalog_manager.cc:1347] Generated new cluster ID: 58be6e30048349f2a0f25085abe98621
I20260926 21:00:01.828330 21108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:00:01.830870 21109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:01.825361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53115 (local address 127.0.0.1:43419)
0926 21:00:01.825764 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:01.825775 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:01.825812 (+ 37us) server_negotiation.cc:407] Connection header received
0926 21:00:01.825937 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:01.825945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:01.826011 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:01.826195 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:01.826438 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:01.827879 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:01.828865 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:01.829302 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:01.829437 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:01.830191 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:01.830195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:01.830196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:01.830379 (+ 183us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:01.830410 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:01.830422 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:01.830506 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:01.830579 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 21:00:01.830643 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":73,"threads_started":1}
I20260926 21:00:01.831720 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:00:01.836001 21109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:01.833132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53803 (local address 127.0.0.1:43419)
0926 21:00:01.833167 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:01.833169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:01.833311 (+ 142us) server_negotiation.cc:407] Connection header received
0926 21:00:01.833382 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:01.833383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:01.833404 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:01.833459 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:01.833591 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:01.834072 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:01.834783 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:01.834996 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:01.835083 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:01.835750 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:01.835751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:01.835751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:01.835756 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:01.835770 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:01.835774 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:01.835822 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:01.835879 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 21:00:01.835903 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 21:00:01.846535 21108 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:00:01.847352 21108 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:00:01.851750 21108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc0d7490454946a3a1cfc3ef8a94691a: Generated new TSK 0
I20260926 21:00:01.851974 21108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:00:01.854737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 21027
2026-09-26T21:00:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:00:01 dist-test-slave-fl7g krb5kdc[21126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:00:01 dist-test-slave-fl7g krb5kdc[21126](info): set up 2 sockets
Sep 26 21:00:01 dist-test-slave-fl7g krb5kdc[21126](info): commencing operation
krb5kdc: starting...
W20260926 21:00:02.900592 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 21:00:02 dist-test-slave-fl7g krb5kdc[21126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:00:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:00:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:00:03.032397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:00:03.115304 21139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:03.115528 21139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:03.115553 21139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:03.116808 21139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:03.116861 21139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:03.116876 21139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:03.116889 21139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:03.116902 21139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:03.118458 21139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:03.118736 21139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:03.118983 21139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:03.124257 21147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:03.124307 21139 server_base.cc:1034] running on GCE node
W20260926 21:00:03.124270 21145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:03.124277 21144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:03.124667 21139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:03.124994 21139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:03.126142 21139 hybrid_clock.cc:648] HybridClock initialized: now 1790456403126124 us; error 41 us; skew 500 ppm
Sep 26 21:00:03 dist-test-slave-fl7g krb5kdc[21126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:03.128695 21139 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:00:03.129184 21139 webserver.cc:460] Webserver started at http://127.0.0.1:36479/ using document root <none> and password file <none>
I20260926 21:00:03.129422 21139 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:03.129491 21139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:03.129664 21139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:03.130760 21139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "2b099f1d76b140bdafe6f903debae406"
format_stamp: "Formatted at 2026-09-26 21:00:03 on dist-test-slave-fl7g"
server_key: "343f43d809d39e7964c62dd7d1552f2e"
server_key_iv: "0881284a85ef33471587ab967416aee8"
server_key_version: "encryptionkey@0"
I20260926 21:00:03.131151 21139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "2b099f1d76b140bdafe6f903debae406"
format_stamp: "Formatted at 2026-09-26 21:00:03 on dist-test-slave-fl7g"
server_key: "343f43d809d39e7964c62dd7d1552f2e"
server_key_iv: "0881284a85ef33471587ab967416aee8"
server_key_version: "encryptionkey@0"
I20260926 21:00:03.132644 21139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 21:00:03.133504 21154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:03.133685 21139 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:00:03.133798 21139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "2b099f1d76b140bdafe6f903debae406"
format_stamp: "Formatted at 2026-09-26 21:00:03 on dist-test-slave-fl7g"
server_key: "343f43d809d39e7964c62dd7d1552f2e"
server_key_iv: "0881284a85ef33471587ab967416aee8"
server_key_version: "encryptionkey@0"
I20260926 21:00:03.133880 21139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:03.137374 21139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:03.139374 21139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:03.139539 21139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:03.143882 21139 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38411
I20260926 21:00:03.143914 21206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38411 every 8 connection(s)
I20260926 21:00:03.144271 21139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 21:00:03.145336 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:03.147648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 21139
I20260926 21:00:03.147778 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 21:00:03.147823 21207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406: Bootstrap starting.
I20260926 21:00:03.147953 392 external_mini_cluster.cc:1428] Setting key 1e1569f223f9b4534eec07fdfb7f0504
I20260926 21:00:03.148602 21207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:03.148962 21207 log.cc:824] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:03.149757 21207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406: No bootstrap required, opened a new log
I20260926 21:00:03.151028 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:03.148545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32823 (local address 127.0.0.1:38411)
0926 21:00:03.148869 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:03.148880 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:03.148916 (+ 36us) server_negotiation.cc:407] Connection header received
0926 21:00:03.149577 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:03.149587 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:03.149624 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:03.149784 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:03.150574 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:03.150579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:03.150579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:03.150584 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:03.150629 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:03.150644 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:03.150742 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:03.150772 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 21:00:03.150813 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":152,"threads_started":1}
I20260926 21:00:03.152251 21207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b099f1d76b140bdafe6f903debae406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } }
I20260926 21:00:03.152403 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:00:03.152462 21207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:00:03.152499 21207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b099f1d76b140bdafe6f903debae406, State: Initialized, Role: FOLLOWER
I20260926 21:00:03.152623 21207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b099f1d76b140bdafe6f903debae406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } }
I20260926 21:00:03.152710 21207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:00:03.152740 21207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:00:03.152776 21207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:03.153610 21207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b099f1d76b140bdafe6f903debae406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } }
I20260926 21:00:03.153740 21207 leader_election.cc:302] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b099f1d76b140bdafe6f903debae406; no voters:
I20260926 21:00:03.153928 21207 leader_election.cc:288] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:00:03.153998 21216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:00:03.154196 21216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406 [term 1 LEADER]: Becoming Leader. State: Replica: 2b099f1d76b140bdafe6f903debae406, State: Running, Role: LEADER
I20260926 21:00:03.154204 21207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:03.154987 21216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b099f1d76b140bdafe6f903debae406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } }
I20260926 21:00:03.155578 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:03.154080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52429 (local address 127.0.0.1:38411)
0926 21:00:03.154118 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:03.154120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:03.154420 (+ 300us) server_negotiation.cc:407] Connection header received
0926 21:00:03.154498 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:03.154499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:03.154505 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:03.154555 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:03.155308 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:03.155309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:03.155309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:03.155313 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:03.155330 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:03.155334 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:03.155396 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:03.155506 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 21:00:03.155520 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 21:00:03.155882 21217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b099f1d76b140bdafe6f903debae406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b099f1d76b140bdafe6f903debae406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } } }
I20260926 21:00:03.155709 21218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b099f1d76b140bdafe6f903debae406. Latest consensus state: current_term: 1 leader_uuid: "2b099f1d76b140bdafe6f903debae406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b099f1d76b140bdafe6f903debae406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } } }
I20260926 21:00:03.155968 21217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406 [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:03.155979 21218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406 [sys.catalog]: This master's current role is: LEADER
W20260926 21:00:03.156545 21231 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:00:03.156611 21231 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:00:03.156667 21232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:00:03.156872 21232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:00:03.158481 21232 catalog_manager.cc:1347] Generated new cluster ID: 922a103615664a9284d21857aedf3a41
I20260926 21:00:03.158553 21232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:00:03.187299 21232 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:00:03.187532 21232 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:00:03.191959 21232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b099f1d76b140bdafe6f903debae406: Generated new TSK 0
I20260926 21:00:03.192196 21232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:00:03.200377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 21139
2026-09-26T21:00:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:00:03 dist-test-slave-fl7g krb5kdc[21238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:00:03 dist-test-slave-fl7g krb5kdc[21238](info): set up 2 sockets
Sep 26 21:00:03 dist-test-slave-fl7g krb5kdc[21238](info): commencing operation
krb5kdc: starting...
W20260926 21:00:04.248767 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 21:00:04 dist-test-slave-fl7g krb5kdc[21238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:00:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:00:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:00:04.377477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41979
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:00:04.463176 21251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:04.463477 21251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:04.463519 21251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:04.464847 21251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:04.464942 21251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:04.464967 21251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:04.464989 21251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:04.465019 21251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:04.466666 21251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:04.466992 21251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:04.467253 21251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:04.472625 21259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:04.472687 21257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:04.472625 21256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:04.472936 21251 server_base.cc:1034] running on GCE node
I20260926 21:00:04.473191 21251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:04.473500 21251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:04.474684 21251 hybrid_clock.cc:648] HybridClock initialized: now 1790456404474661 us; error 38 us; skew 500 ppm
Sep 26 21:00:04 dist-test-slave-fl7g krb5kdc[21238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:04.477406 21251 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:00:04.477914 21251 webserver.cc:460] Webserver started at http://127.0.0.1:42247/ using document root <none> and password file <none>
I20260926 21:00:04.478148 21251 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:04.478219 21251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:04.478375 21251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:04.479494 21251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "4ec5a97016654aba8cb01d9e27d12fe3"
format_stamp: "Formatted at 2026-09-26 21:00:04 on dist-test-slave-fl7g"
server_key: "54fc988181a5ecd07925a99ce9a06526"
server_key_iv: "b80b32cafdb7e0a209ae7764b9191a05"
server_key_version: "encryptionkey@0"
I20260926 21:00:04.479887 21251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "4ec5a97016654aba8cb01d9e27d12fe3"
format_stamp: "Formatted at 2026-09-26 21:00:04 on dist-test-slave-fl7g"
server_key: "54fc988181a5ecd07925a99ce9a06526"
server_key_iv: "b80b32cafdb7e0a209ae7764b9191a05"
server_key_version: "encryptionkey@0"
I20260926 21:00:04.481504 21251 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 21:00:04.482476 21266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:04.482712 21251 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 21:00:04.482851 21251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "4ec5a97016654aba8cb01d9e27d12fe3"
format_stamp: "Formatted at 2026-09-26 21:00:04 on dist-test-slave-fl7g"
server_key: "54fc988181a5ecd07925a99ce9a06526"
server_key_iv: "b80b32cafdb7e0a209ae7764b9191a05"
server_key_version: "encryptionkey@0"
I20260926 21:00:04.482939 21251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:04.490061 21251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:04.492319 21251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:04.492524 21251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:04.496886 21251 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41979
I20260926 21:00:04.496955 21318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41979 every 8 connection(s)
I20260926 21:00:04.497386 21251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 21:00:04.498366 21319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:04.500923 21319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3: Bootstrap starting.
I20260926 21:00:04.501541 21319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:04.501835 21319 log.cc:824] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:04.502596 21319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3: No bootstrap required, opened a new log
I20260926 21:00:04.503141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 21251
I20260926 21:00:04.503265 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 21:00:04.503468 392 external_mini_cluster.cc:1428] Setting key 7ed6b2abab8fc6fa530f83b6c38a4f0c
I20260926 21:00:04.505203 21319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec5a97016654aba8cb01d9e27d12fe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41979 } }
I20260926 21:00:04.505445 21319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:00:04.505481 21319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec5a97016654aba8cb01d9e27d12fe3, State: Initialized, Role: FOLLOWER
I20260926 21:00:04.505674 21319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec5a97016654aba8cb01d9e27d12fe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41979 } }
I20260926 21:00:04.505839 21319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:00:04.505909 21319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:00:04.505972 21319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:04.506609 21319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec5a97016654aba8cb01d9e27d12fe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41979 } }
I20260926 21:00:04.506752 21319 leader_election.cc:302] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ec5a97016654aba8cb01d9e27d12fe3; no voters:
I20260926 21:00:04.506990 21319 leader_election.cc:288] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:00:04.507076 21324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:00:04.507274 21324 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3 [term 1 LEADER]: Becoming Leader. State: Replica: 4ec5a97016654aba8cb01d9e27d12fe3, State: Running, Role: LEADER
I20260926 21:00:04.507274 21319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:04.507448 21324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec5a97016654aba8cb01d9e27d12fe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41979 } }
I20260926 21:00:04.507889 21325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ec5a97016654aba8cb01d9e27d12fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec5a97016654aba8cb01d9e27d12fe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41979 } } }
I20260926 21:00:04.507972 21325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3 [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:04.508158 21326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ec5a97016654aba8cb01d9e27d12fe3. Latest consensus state: current_term: 1 leader_uuid: "4ec5a97016654aba8cb01d9e27d12fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec5a97016654aba8cb01d9e27d12fe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41979 } } }
I20260926 21:00:04.508210 21326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3 [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:04.508638 21333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:00:04.509195 21333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:00:04.511454 21333 catalog_manager.cc:1347] Generated new cluster ID: 29614a01877d47f4bede17cb74237ac6
I20260926 21:00:04.511521 21333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 21:00:04 dist-test-slave-fl7g krb5kdc[21238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 21:00:04.517503 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:04.504065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.174:44977 (local address 127.0.0.1:41979)
0926 21:00:04.504312 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:04.504323 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:04.504362 (+ 39us) server_negotiation.cc:407] Connection header received
0926 21:00:04.505406 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:04.505417 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:04.505493 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:04.505664 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:04.506880 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:04.508442 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:04.509560 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:04.509988 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:04.510118 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:04.513494 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:04.513499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:04.513499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:04.513516 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:04.515414 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:04.515878 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:04.515887 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:04.515887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:04.515983 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:04.516229 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:04.516230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:04.516230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:04.516611 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:04.517175 (+ 564us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:04.517253 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 21:00:04.517343 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":110,"threads_started":1}
I20260926 21:00:04.518606 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:00:04.526645 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:04.520114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.174:54937 (local address 127.0.0.1:41979)
0926 21:00:04.520162 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:04.520164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:04.520342 (+ 178us) server_negotiation.cc:407] Connection header received
0926 21:00:04.520451 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:04.520452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:04.520465 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:04.520511 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:04.521066 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:04.521593 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:04.522348 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:04.522581 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:04.522662 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:04.523955 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:04.523956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:04.523957 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:04.523965 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:04.525438 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:04.525822 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:04.525823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:04.525824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:04.525887 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:04.526111 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:04.526112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:04.526112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:04.526238 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:04.526351 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:04.526473 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 21:00:04.526537 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 21:00:04.534617 21333 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:00:04.535480 21333 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:00:04.544713 21333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ec5a97016654aba8cb01d9e27d12fe3: Generated new TSK 0
I20260926 21:00:04.544971 21333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:00:04.551442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 21251
W20260926 21:00:04.555775 21236 connection.cc:535] client connection to 127.0.0.1:41979 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41979): Connection reset by peer (error 104)
2026-09-26T21:00:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:00:04 dist-test-slave-fl7g krb5kdc[21350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:00:04 dist-test-slave-fl7g krb5kdc[21350](info): set up 2 sockets
Sep 26 21:00:04 dist-test-slave-fl7g krb5kdc[21350](info): commencing operation
krb5kdc: starting...
W20260926 21:00:05.592661 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 21:00:05 dist-test-slave-fl7g krb5kdc[21350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:00:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:00:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:00:05.719964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46301
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:00:05.803246 21363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:05.803540 21363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:05.803588 21363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:05.804903 21363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:05.804986 21363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:05.805011 21363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:05.805043 21363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:05.805071 21363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:05.806679 21363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:05.806996 21363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:05.807255 21363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:05.812351 21368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:05.812458 21369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:05.812525 21363 server_base.cc:1034] running on GCE node
W20260926 21:00:05.812474 21371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:05.813041 21363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:05.813367 21363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:05.814595 21363 hybrid_clock.cc:648] HybridClock initialized: now 1790456405814569 us; error 51 us; skew 500 ppm
Sep 26 21:00:05 dist-test-slave-fl7g krb5kdc[21350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:05.817593 21363 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:00:05.818128 21363 webserver.cc:460] Webserver started at http://127.0.0.1:34753/ using document root <none> and password file <none>
I20260926 21:00:05.818382 21363 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:05.818459 21363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:05.818609 21363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:05.819749 21363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "4a45eb4109bc49aabeb456e50a5142fc"
format_stamp: "Formatted at 2026-09-26 21:00:05 on dist-test-slave-fl7g"
server_key: "4901fa2d037819897cab20956ed6bd45"
server_key_iv: "04fba3072dc94a4a079cce317d6c42d1"
server_key_version: "encryptionkey@0"
I20260926 21:00:05.820135 21363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a45eb4109bc49aabeb456e50a5142fc"
format_stamp: "Formatted at 2026-09-26 21:00:05 on dist-test-slave-fl7g"
server_key: "4901fa2d037819897cab20956ed6bd45"
server_key_iv: "04fba3072dc94a4a079cce317d6c42d1"
server_key_version: "encryptionkey@0"
I20260926 21:00:05.821696 21363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 21:00:05.822549 21378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:05.822779 21363 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:00:05.822887 21363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "4a45eb4109bc49aabeb456e50a5142fc"
format_stamp: "Formatted at 2026-09-26 21:00:05 on dist-test-slave-fl7g"
server_key: "4901fa2d037819897cab20956ed6bd45"
server_key_iv: "04fba3072dc94a4a079cce317d6c42d1"
server_key_version: "encryptionkey@0"
I20260926 21:00:05.822978 21363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:05.855093 21363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:05.857707 21363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:05.857908 21363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:05.862180 21363 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46301
I20260926 21:00:05.862269 21430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46301 every 8 connection(s)
I20260926 21:00:05.862638 21363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 21:00:05.863626 21431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:05.865763 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 21363
I20260926 21:00:05.865947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 21:00:05.866149 392 external_mini_cluster.cc:1428] Setting key 632bd007295233a356810abf44fc976f
I20260926 21:00:05.866236 21431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc: Bootstrap starting.
I20260926 21:00:05.867045 21431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:05.867352 21431 log.cc:824] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:05.868185 21431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc: No bootstrap required, opened a new log
I20260926 21:00:05.870529 21431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a45eb4109bc49aabeb456e50a5142fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46301 } }
I20260926 21:00:05.870680 21431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:00:05.870703 21431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a45eb4109bc49aabeb456e50a5142fc, State: Initialized, Role: FOLLOWER
I20260926 21:00:05.870862 21431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a45eb4109bc49aabeb456e50a5142fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46301 } }
I20260926 21:00:05.870954 21431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:00:05.870988 21431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:00:05.871093 21431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:05.871851 21431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a45eb4109bc49aabeb456e50a5142fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46301 } }
I20260926 21:00:05.871945 21431 leader_election.cc:302] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a45eb4109bc49aabeb456e50a5142fc; no voters:
I20260926 21:00:05.872159 21431 leader_election.cc:288] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:00:05.872185 21436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:00:05.872428 21436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc [term 1 LEADER]: Becoming Leader. State: Replica: 4a45eb4109bc49aabeb456e50a5142fc, State: Running, Role: LEADER
I20260926 21:00:05.872563 21436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a45eb4109bc49aabeb456e50a5142fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46301 } }
I20260926 21:00:05.872538 21431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:05.872550 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:05.866767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.174:50279 (local address 127.0.0.1:46301)
0926 21:00:05.866966 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:05.866974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:05.867001 (+ 27us) server_negotiation.cc:407] Connection header received
0926 21:00:05.867594 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:05.867604 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:05.867677 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:05.867850 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:05.868101 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:05.869137 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:05.870337 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:05.870872 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:05.871000 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:05.871679 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:05.871682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:05.871683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:05.871885 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:05.871953 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:05.871971 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:05.872058 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:05.872210 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 21:00:05.872266 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260926 21:00:05.873085 21437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a45eb4109bc49aabeb456e50a5142fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a45eb4109bc49aabeb456e50a5142fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46301 } } }
I20260926 21:00:05.873185 21437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:05.873083 21438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a45eb4109bc49aabeb456e50a5142fc. Latest consensus state: current_term: 1 leader_uuid: "4a45eb4109bc49aabeb456e50a5142fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a45eb4109bc49aabeb456e50a5142fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46301 } } }
I20260926 21:00:05.873332 21438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:05.873548 21441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:00:05.874060 21441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:00:05.874109 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:00:05.875838 21441 catalog_manager.cc:1347] Generated new cluster ID: ca5771edc22d41afb00b09f44f1754a6
I20260926 21:00:05.875895 21441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:00:05.879498 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:05.875718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.174:60315 (local address 127.0.0.1:46301)
0926 21:00:05.875759 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:05.875762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:05.875957 (+ 195us) server_negotiation.cc:407] Connection header received
0926 21:00:05.876046 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:05.876047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:05.876058 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:05.876119 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:05.876274 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:05.877039 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:05.878099 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:05.878410 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:05.878470 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:05.879213 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:05.879214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:05.879214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:05.879219 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:05.879242 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:05.879247 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:05.879302 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:05.879406 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 21:00:05.879418 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 21:00:05.881927 21441 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:00:05.882702 21441 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:00:05.896543 21441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a45eb4109bc49aabeb456e50a5142fc: Generated new TSK 0
I20260926 21:00:05.896773 21441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:00:05.900378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 21363
W20260926 21:00:05.904301 21348 connection.cc:535] client connection to 127.0.0.1:46301 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46301): Connection reset by peer (error 104)
2026-09-26T21:00:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:00:05 dist-test-slave-fl7g krb5kdc[21462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:00:05 dist-test-slave-fl7g krb5kdc[21462](info): set up 2 sockets
Sep 26 21:00:05 dist-test-slave-fl7g krb5kdc[21462](info): commencing operation
krb5kdc: starting...
W20260926 21:00:06.956969 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 21:00:07 dist-test-slave-fl7g krb5kdc[21462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:00:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:00:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:00:07.088311 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:00:07.174530 21475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:07.174806 21475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:07.174849 21475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:07.176096 21475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:07.176185 21475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:07.176209 21475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:07.176234 21475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:07.176263 21475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:07.177970 21475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:07.178292 21475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:07.178541 21475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:07.183892 21480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:07.183911 21475 server_base.cc:1034] running on GCE node
W20260926 21:00:07.183924 21481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:07.183894 21483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:07.184507 21475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:07.184880 21475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:07.186053 21475 hybrid_clock.cc:648] HybridClock initialized: now 1790456407186034 us; error 37 us; skew 500 ppm
Sep 26 21:00:07 dist-test-slave-fl7g krb5kdc[21462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:07.192515 21475 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:00:07.193128 21475 webserver.cc:460] Webserver started at http://127.0.0.1:33741/ using document root <none> and password file <none>
I20260926 21:00:07.193379 21475 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:07.193454 21475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:07.193627 21475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:07.194756 21475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "5b918cca586f4196ba024ba7c3478932"
format_stamp: "Formatted at 2026-09-26 21:00:07 on dist-test-slave-fl7g"
server_key: "ceac1a32e93987d7d2e770e2ae4c77d6"
server_key_iv: "431745d6ca822e9ae3a98ee68e6e2881"
server_key_version: "encryptionkey@0"
I20260926 21:00:07.195166 21475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "5b918cca586f4196ba024ba7c3478932"
format_stamp: "Formatted at 2026-09-26 21:00:07 on dist-test-slave-fl7g"
server_key: "ceac1a32e93987d7d2e770e2ae4c77d6"
server_key_iv: "431745d6ca822e9ae3a98ee68e6e2881"
server_key_version: "encryptionkey@0"
I20260926 21:00:07.196741 21475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 21:00:07.197650 21490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:07.197909 21475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:00:07.198033 21475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "5b918cca586f4196ba024ba7c3478932"
format_stamp: "Formatted at 2026-09-26 21:00:07 on dist-test-slave-fl7g"
server_key: "ceac1a32e93987d7d2e770e2ae4c77d6"
server_key_iv: "431745d6ca822e9ae3a98ee68e6e2881"
server_key_version: "encryptionkey@0"
I20260926 21:00:07.198119 21475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:07.201691 21475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:07.203852 21475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:07.204021 21475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:07.208019 21475 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37329
I20260926 21:00:07.208159 21542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37329 every 8 connection(s)
I20260926 21:00:07.208513 21475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 21:00:07.209638 21543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:07.212195 21543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932: Bootstrap starting.
I20260926 21:00:07.212918 21543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:07.213238 21543 log.cc:824] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:07.213483 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 21475
I20260926 21:00:07.213603 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 21:00:07.213815 392 external_mini_cluster.cc:1428] Setting key e4863018c313adfdf8cd5ac884665dfc
I20260926 21:00:07.214184 21543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932: No bootstrap required, opened a new log
I20260926 21:00:07.216565 21543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b918cca586f4196ba024ba7c3478932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37329 } }
I20260926 21:00:07.216746 21543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:00:07.216804 21543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b918cca586f4196ba024ba7c3478932, State: Initialized, Role: FOLLOWER
I20260926 21:00:07.216955 21543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b918cca586f4196ba024ba7c3478932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37329 } }
I20260926 21:00:07.217060 21543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:00:07.217089 21543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:00:07.217164 21543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:07.217291 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:07.214388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.174:34529 (local address 127.0.0.1:37329)
0926 21:00:07.214591 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:07.214603 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:07.214635 (+ 32us) server_negotiation.cc:407] Connection header received
0926 21:00:07.215522 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:07.215530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:07.215560 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:07.215755 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:07.216518 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:07.216522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:07.216522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:07.216526 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:07.216574 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:07.216585 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:07.216665 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:07.216879 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 21:00:07.216929 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":83,"threads_started":1}
I20260926 21:00:07.218102 21543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b918cca586f4196ba024ba7c3478932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37329 } }
I20260926 21:00:07.218232 21543 leader_election.cc:302] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b918cca586f4196ba024ba7c3478932; no voters:
I20260926 21:00:07.218434 21543 leader_election.cc:288] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:00:07.218441 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:00:07.218463 21548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:00:07.218595 21548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932 [term 1 LEADER]: Becoming Leader. State: Replica: 5b918cca586f4196ba024ba7c3478932, State: Running, Role: LEADER
I20260926 21:00:07.218731 21543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:07.218736 21548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b918cca586f4196ba024ba7c3478932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37329 } }
I20260926 21:00:07.219229 21549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b918cca586f4196ba024ba7c3478932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b918cca586f4196ba024ba7c3478932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37329 } } }
I20260926 21:00:07.219244 21550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b918cca586f4196ba024ba7c3478932. Latest consensus state: current_term: 1 leader_uuid: "5b918cca586f4196ba024ba7c3478932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b918cca586f4196ba024ba7c3478932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37329 } } }
I20260926 21:00:07.219480 21550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932 [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:07.219753 21559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:00:07.219376 21549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932 [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:07.220435 21559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:00:07.222018 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:07.220105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.174:35769 (local address 127.0.0.1:37329)
0926 21:00:07.220142 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:07.220144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:07.220803 (+ 659us) server_negotiation.cc:407] Connection header received
0926 21:00:07.220905 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:07.220906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:07.220914 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:07.220973 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:07.221740 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:07.221741 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:07.221741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:07.221746 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:07.221761 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:07.221765 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:07.221797 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:07.221928 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 21:00:07.221944 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 21:00:07.222815 21559 catalog_manager.cc:1347] Generated new cluster ID: e60b6a0a59fc4804b1971d7b7bfe1d13
I20260926 21:00:07.222892 21559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:00:07.232306 21559 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:00:07.232570 21559 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:00:07.245051 21559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b918cca586f4196ba024ba7c3478932: Generated new TSK 0
I20260926 21:00:07.245299 21559 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:00:07.252094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 21475
2026-09-26T21:00:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:00:07 dist-test-slave-fl7g krb5kdc[21574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:00:07 dist-test-slave-fl7g krb5kdc[21574](info): set up 2 sockets
Sep 26 21:00:07 dist-test-slave-fl7g krb5kdc[21574](info): commencing operation
krb5kdc: starting...
W20260926 21:00:08.292685 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 21:00:08 dist-test-slave-fl7g krb5kdc[21574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:00:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:00:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:00:08.423415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:00:08.506513 21587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:08.506783 21587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:08.506824 21587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:08.508160 21587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:08.508226 21587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:08.508249 21587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:08.508275 21587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:08.508306 21587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:08.509920 21587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45667
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:08.510233 21587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:08.510473 21587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:08.515548 21593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:08.515530 21592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:08.515600 21595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:08.515764 21587 server_base.cc:1034] running on GCE node
I20260926 21:00:08.516000 21587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:08.516295 21587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:08.517480 21587 hybrid_clock.cc:648] HybridClock initialized: now 1790456408517468 us; error 46 us; skew 500 ppm
Sep 26 21:00:08 dist-test-slave-fl7g krb5kdc[21574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:08.519928 21587 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:00:08.520540 21587 webserver.cc:460] Webserver started at http://127.0.0.1:44321/ using document root <none> and password file <none>
I20260926 21:00:08.520792 21587 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:08.520864 21587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:08.521003 21587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:08.522113 21587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "6de478e033844016bd48a8902c788efb"
format_stamp: "Formatted at 2026-09-26 21:00:08 on dist-test-slave-fl7g"
server_key: "4554c793f9f46ac34b7d2e74388ac9a5"
server_key_iv: "66e46f9c3d2db42d5a501cb3de8cdb85"
server_key_version: "encryptionkey@0"
I20260926 21:00:08.522505 21587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "6de478e033844016bd48a8902c788efb"
format_stamp: "Formatted at 2026-09-26 21:00:08 on dist-test-slave-fl7g"
server_key: "4554c793f9f46ac34b7d2e74388ac9a5"
server_key_iv: "66e46f9c3d2db42d5a501cb3de8cdb85"
server_key_version: "encryptionkey@0"
I20260926 21:00:08.524031 21587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 21:00:08.525065 21602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:08.525336 21587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 21:00:08.525475 21587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "6de478e033844016bd48a8902c788efb"
format_stamp: "Formatted at 2026-09-26 21:00:08 on dist-test-slave-fl7g"
server_key: "4554c793f9f46ac34b7d2e74388ac9a5"
server_key_iv: "66e46f9c3d2db42d5a501cb3de8cdb85"
server_key_version: "encryptionkey@0"
I20260926 21:00:08.525570 21587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:08.545583 21587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:08.547894 21587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:08.548074 21587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:08.552590 21587 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45667
I20260926 21:00:08.552744 21654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45667 every 8 connection(s)
I20260926 21:00:08.553054 21587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 21:00:08.553982 21655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:08.556253 21655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb: Bootstrap starting.
I20260926 21:00:08.556911 21655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:08.557165 21655 log.cc:824] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:08.557917 21655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb: No bootstrap required, opened a new log
I20260926 21:00:08.558916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 21587
I20260926 21:00:08.559046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 21:00:08.559234 392 external_mini_cluster.cc:1428] Setting key 6f7eedb9d3de40e96157045e12a0e38f
I20260926 21:00:08.559837 21655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6de478e033844016bd48a8902c788efb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45667 } }
I20260926 21:00:08.560032 21655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:00:08.560071 21655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6de478e033844016bd48a8902c788efb, State: Initialized, Role: FOLLOWER
I20260926 21:00:08.560215 21655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6de478e033844016bd48a8902c788efb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45667 } }
I20260926 21:00:08.560297 21655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:00:08.560325 21655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:00:08.560377 21655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:08.561529 21655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6de478e033844016bd48a8902c788efb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45667 } }
I20260926 21:00:08.561650 21655 leader_election.cc:302] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6de478e033844016bd48a8902c788efb; no voters:
I20260926 21:00:08.561821 21655 leader_election.cc:288] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:00:08.561937 21660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:00:08.562072 21655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:08.562079 21660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb [term 1 LEADER]: Becoming Leader. State: Replica: 6de478e033844016bd48a8902c788efb, State: Running, Role: LEADER
I20260926 21:00:08.562282 21660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6de478e033844016bd48a8902c788efb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45667 } }
I20260926 21:00:08.562613 21660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6de478e033844016bd48a8902c788efb. Latest consensus state: current_term: 1 leader_uuid: "6de478e033844016bd48a8902c788efb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6de478e033844016bd48a8902c788efb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45667 } } }
I20260926 21:00:08.562685 21660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:08.562788 21662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6de478e033844016bd48a8902c788efb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6de478e033844016bd48a8902c788efb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45667 } } }
I20260926 21:00:08.562917 21662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:08.563027 21669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:00:08.563532 21669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:00:08.566327 21669 catalog_manager.cc:1347] Generated new cluster ID: 39f86941383048feb62ab0ae9e5d7e57
I20260926 21:00:08.566387 21669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 21:00:08 dist-test-slave-fl7g krb5kdc[21574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 21:00:08.570760 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:08.559826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.174:60891 (local address 127.0.0.1:45667)
0926 21:00:08.560060 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:08.560071 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:08.560107 (+ 36us) server_negotiation.cc:407] Connection header received
0926 21:00:08.560225 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:08.560233 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:08.560292 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:08.560477 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:08.561110 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:08.562567 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:08.563780 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:08.564385 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:08.564538 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:08.567275 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:08.567279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:08.567279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:08.567294 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:08.569121 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:08.569529 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:08.569539 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:08.569540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:08.569647 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:08.569899 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:08.569900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:08.569900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:08.570169 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:08.570349 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:08.570475 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 21:00:08.570587 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":122,"threads_started":1}
I20260926 21:00:08.571831 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:00:08.579777 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:08.573311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.174:40133 (local address 127.0.0.1:45667)
0926 21:00:08.573353 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:08.573355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:08.573493 (+ 138us) server_negotiation.cc:407] Connection header received
0926 21:00:08.573550 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:08.573551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:08.573562 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:08.573616 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:08.574148 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:08.574662 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:08.575398 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:08.575637 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:08.575722 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:08.577075 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:08.577077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:08.577077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:08.577086 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:08.578565 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:08.578953 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:08.578954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:08.578955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:08.579013 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:08.579243 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:08.579243 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:08.579243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:08.579360 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:08.579489 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:08.579599 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 21:00:08.579663 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 21:00:08.609090 21669 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:00:08.609665 21669 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:00:08.613909 21669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6de478e033844016bd48a8902c788efb: Generated new TSK 0
I20260926 21:00:08.614121 21669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:00:08.621726 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 21587
W20260926 21:00:08.626026 21572 connection.cc:535] client connection to 127.0.0.1:45667 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45667): Connection reset by peer (error 104)
2026-09-26T21:00:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:00:08 dist-test-slave-fl7g krb5kdc[21686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:00:08 dist-test-slave-fl7g krb5kdc[21686](info): set up 2 sockets
Sep 26 21:00:08 dist-test-slave-fl7g krb5kdc[21686](info): commencing operation
krb5kdc: starting...
W20260926 21:00:09.664683 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 21:00:09 dist-test-slave-fl7g krb5kdc[21686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:00:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:00:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:00:09.792745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:00:09.875717 21699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:09.875993 21699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:09.876037 21699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:09.877374 21699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:09.877454 21699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:09.877483 21699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:09.877538 21699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:09.877569 21699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:09.879161 21699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37879
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:09.879477 21699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:09.879738 21699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:09.885031 21704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:09.885025 21707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:09.885048 21705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:09.885087 21699 server_base.cc:1034] running on GCE node
I20260926 21:00:09.885447 21699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:09.885779 21699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:09.886969 21699 hybrid_clock.cc:648] HybridClock initialized: now 1790456409886935 us; error 51 us; skew 500 ppm
Sep 26 21:00:09 dist-test-slave-fl7g krb5kdc[21686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:09.889752 21699 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:00:09.890244 21699 webserver.cc:460] Webserver started at http://127.0.0.1:44227/ using document root <none> and password file <none>
I20260926 21:00:09.890499 21699 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:09.890549 21699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:09.890681 21699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:09.891770 21699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "254f5d1e99ca4a36a7249104ffe04781"
format_stamp: "Formatted at 2026-09-26 21:00:09 on dist-test-slave-fl7g"
server_key: "413d92fdfc222d81daf851aadde1b690"
server_key_iv: "926fb975fc497795447e5b66b88be2d8"
server_key_version: "encryptionkey@0"
I20260926 21:00:09.892127 21699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "254f5d1e99ca4a36a7249104ffe04781"
format_stamp: "Formatted at 2026-09-26 21:00:09 on dist-test-slave-fl7g"
server_key: "413d92fdfc222d81daf851aadde1b690"
server_key_iv: "926fb975fc497795447e5b66b88be2d8"
server_key_version: "encryptionkey@0"
I20260926 21:00:09.893607 21699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260926 21:00:09.894506 21714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:09.894748 21699 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:00:09.894837 21699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "254f5d1e99ca4a36a7249104ffe04781"
format_stamp: "Formatted at 2026-09-26 21:00:09 on dist-test-slave-fl7g"
server_key: "413d92fdfc222d81daf851aadde1b690"
server_key_iv: "926fb975fc497795447e5b66b88be2d8"
server_key_version: "encryptionkey@0"
I20260926 21:00:09.894897 21699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:09.908013 21699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:09.910410 21699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:09.910550 21699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:09.914871 21699 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37879
I20260926 21:00:09.914973 21766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37879 every 8 connection(s)
I20260926 21:00:09.915323 21699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 21:00:09.916466 21767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:09.917999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 21699
I20260926 21:00:09.918102 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 21:00:09.918331 392 external_mini_cluster.cc:1428] Setting key 6b17b8d7d60807abf0d27b80f7cb9cba
I20260926 21:00:09.919106 21767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781: Bootstrap starting.
I20260926 21:00:09.919842 21767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:09.920166 21767 log.cc:824] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:09.920907 21767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781: No bootstrap required, opened a new log
I20260926 21:00:09.922679 21767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254f5d1e99ca4a36a7249104ffe04781" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260926 21:00:09.922844 21767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:00:09.922896 21767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254f5d1e99ca4a36a7249104ffe04781, State: Initialized, Role: FOLLOWER
I20260926 21:00:09.923331 21767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254f5d1e99ca4a36a7249104ffe04781" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260926 21:00:09.923424 21767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:00:09.923449 21767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:00:09.923518 21767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:09.924096 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:09.918937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.174:47879 (local address 127.0.0.1:37879)
0926 21:00:09.919149 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:09.919171 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:09.919200 (+ 29us) server_negotiation.cc:407] Connection header received
0926 21:00:09.919782 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:09.919792 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:09.919863 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:09.920062 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:09.920320 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:09.921374 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:09.922182 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:09.922596 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:09.922735 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:09.923384 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:09.923387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:09.923388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:09.923619 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:09.923660 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:09.923671 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:09.923802 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:09.923901 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 21:00:09.923932 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":76,"threads_started":1}
I20260926 21:00:09.924213 21767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254f5d1e99ca4a36a7249104ffe04781" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260926 21:00:09.924321 21767 leader_election.cc:302] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 254f5d1e99ca4a36a7249104ffe04781; no voters:
I20260926 21:00:09.924551 21767 leader_election.cc:288] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:00:09.924609 21772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:00:09.924759 21772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781 [term 1 LEADER]: Becoming Leader. State: Replica: 254f5d1e99ca4a36a7249104ffe04781, State: Running, Role: LEADER
I20260926 21:00:09.924808 21767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:09.924922 21772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254f5d1e99ca4a36a7249104ffe04781" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260926 21:00:09.925277 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:00:09.925318 21773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "254f5d1e99ca4a36a7249104ffe04781" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254f5d1e99ca4a36a7249104ffe04781" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } } }
I20260926 21:00:09.925379 21773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781 [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:09.925516 21774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 254f5d1e99ca4a36a7249104ffe04781. Latest consensus state: current_term: 1 leader_uuid: "254f5d1e99ca4a36a7249104ffe04781" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254f5d1e99ca4a36a7249104ffe04781" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } } }
I20260926 21:00:09.925555 21774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781 [sys.catalog]: This master's current role is: LEADER
W20260926 21:00:09.926637 21788 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:00:09.927001 21788 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:00:09.927109 21779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:00:09.927327 21779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:00:09.928958 21779 catalog_manager.cc:1347] Generated new cluster ID: ebc14289c5fd42868b606dd73acc3ed2
I20260926 21:00:09.929016 21779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:00:09.934377 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:09.926882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.174:56457 (local address 127.0.0.1:37879)
0926 21:00:09.926943 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:09.926945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:09.927290 (+ 345us) server_negotiation.cc:407] Connection header received
0926 21:00:09.927312 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:09.927312 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:09.927325 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:09.927379 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:09.927582 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:09.928242 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:09.932925 (+ 4683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:09.933265 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:09.933346 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:09.934085 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:09.934087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:09.934087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:09.934093 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:09.934111 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:09.934116 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:09.934186 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:09.934279 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 21:00:09.934291 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 21:00:09.935838 21779 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:00:09.936347 21779 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:00:09.942919 21779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 254f5d1e99ca4a36a7249104ffe04781: Generated new TSK 0
I20260926 21:00:09.943143 21779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:00:09.950861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 21699
W20260926 21:00:09.954947 21684 connection.cc:535] client connection to 127.0.0.1:37879 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37879): Connection reset by peer (error 104)
2026-09-26T21:00:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:00:10 dist-test-slave-fl7g krb5kdc[21798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:00:10 dist-test-slave-fl7g krb5kdc[21798](info): set up 2 sockets
Sep 26 21:00:10 dist-test-slave-fl7g krb5kdc[21798](info): commencing operation
krb5kdc: starting...
W20260926 21:00:11.004668 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 21:00:11 dist-test-slave-fl7g krb5kdc[21798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:00:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:00:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:00:11.131470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44789
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:00:11.214987 21811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:11.215286 21811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:11.215332 21811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:11.216712 21811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:11.216794 21811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:11.216820 21811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:11.216848 21811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:11.216878 21811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:11.218487 21811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44789
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:11.218791 21811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:11.219045 21811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:11.224241 21817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:11.224252 21816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:11.224261 21819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:11.224611 21811 server_base.cc:1034] running on GCE node
I20260926 21:00:11.224929 21811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:11.225296 21811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:11.226466 21811 hybrid_clock.cc:648] HybridClock initialized: now 1790456411226451 us; error 49 us; skew 500 ppm
Sep 26 21:00:11 dist-test-slave-fl7g krb5kdc[21798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:11.229260 21811 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:00:11.229766 21811 webserver.cc:460] Webserver started at http://127.0.0.1:36019/ using document root <none> and password file <none>
I20260926 21:00:11.230036 21811 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:11.230111 21811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:11.230280 21811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:11.231420 21811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "6af39906e9064d3dbc3d65c38ccbfd0f"
format_stamp: "Formatted at 2026-09-26 21:00:11 on dist-test-slave-fl7g"
server_key: "c50156108475479ddbdec9471b418129"
server_key_iv: "b53a1465ac001840e3d3294369cde315"
server_key_version: "encryptionkey@0"
I20260926 21:00:11.231815 21811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "6af39906e9064d3dbc3d65c38ccbfd0f"
format_stamp: "Formatted at 2026-09-26 21:00:11 on dist-test-slave-fl7g"
server_key: "c50156108475479ddbdec9471b418129"
server_key_iv: "b53a1465ac001840e3d3294369cde315"
server_key_version: "encryptionkey@0"
I20260926 21:00:11.233363 21811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 21:00:11.234337 21826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:11.234635 21811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 21:00:11.234812 21811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "6af39906e9064d3dbc3d65c38ccbfd0f"
format_stamp: "Formatted at 2026-09-26 21:00:11 on dist-test-slave-fl7g"
server_key: "c50156108475479ddbdec9471b418129"
server_key_iv: "b53a1465ac001840e3d3294369cde315"
server_key_version: "encryptionkey@0"
I20260926 21:00:11.234905 21811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:11.238550 21811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:11.241040 21811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:11.241216 21811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:11.245301 21811 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44789
I20260926 21:00:11.245330 21878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44789 every 8 connection(s)
I20260926 21:00:11.245716 21811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 21:00:11.246804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 21811
I20260926 21:00:11.246924 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 21:00:11.246968 21879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:11.247120 392 external_mini_cluster.cc:1428] Setting key ef2b7c3aae5f6db7f1f4e36d316bab03
I20260926 21:00:11.250208 21882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:11.247731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.174:49507 (local address 127.0.0.1:44789)
0926 21:00:11.247950 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:11.247960 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:11.247998 (+ 38us) server_negotiation.cc:407] Connection header received
0926 21:00:11.248605 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:11.248614 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:11.248644 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:11.248812 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:11.249731 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:11.249735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:11.249736 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:11.249740 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:11.249798 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:11.249815 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:11.249901 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:11.249978 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 21:00:11.250028 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":86,"threads_started":1}
I20260926 21:00:11.251083 21879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f: Bootstrap starting.
I20260926 21:00:11.251219 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:00:11.251926 21879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:11.252290 21879 log.cc:824] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:11.253221 21879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f: No bootstrap required, opened a new log
I20260926 21:00:11.254305 21882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:11.252992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.174:38211 (local address 127.0.0.1:44789)
0926 21:00:11.253029 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:11.253031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:11.253206 (+ 175us) server_negotiation.cc:407] Connection header received
0926 21:00:11.253238 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:11.253239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:11.253244 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:11.253345 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:11.254062 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:11.254064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:11.254064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:11.254067 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:11.254082 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:11.254085 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:11.254122 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:11.254191 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 21:00:11.254201 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 21:00:11.255703 21879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6af39906e9064d3dbc3d65c38ccbfd0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44789 } }
I20260926 21:00:11.255852 21879 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:00:11.255875 21879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af39906e9064d3dbc3d65c38ccbfd0f, State: Initialized, Role: FOLLOWER
I20260926 21:00:11.256002 21879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6af39906e9064d3dbc3d65c38ccbfd0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44789 } }
I20260926 21:00:11.256094 21879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:00:11.256117 21879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:00:11.256147 21879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:11.256812 21879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6af39906e9064d3dbc3d65c38ccbfd0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44789 } }
I20260926 21:00:11.256930 21879 leader_election.cc:302] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6af39906e9064d3dbc3d65c38ccbfd0f; no voters:
I20260926 21:00:11.257099 21879 leader_election.cc:288] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:00:11.257164 21889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:00:11.257378 21879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:11.257383 21889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f [term 1 LEADER]: Becoming Leader. State: Replica: 6af39906e9064d3dbc3d65c38ccbfd0f, State: Running, Role: LEADER
I20260926 21:00:11.257567 21889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6af39906e9064d3dbc3d65c38ccbfd0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44789 } }
I20260926 21:00:11.257977 21890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6af39906e9064d3dbc3d65c38ccbfd0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6af39906e9064d3dbc3d65c38ccbfd0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44789 } } }
I20260926 21:00:11.258018 21891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6af39906e9064d3dbc3d65c38ccbfd0f. Latest consensus state: current_term: 1 leader_uuid: "6af39906e9064d3dbc3d65c38ccbfd0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6af39906e9064d3dbc3d65c38ccbfd0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44789 } } }
I20260926 21:00:11.258112 21890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:11.258131 21891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:11.258889 21898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:00:11.259435 21898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:00:11.261089 21898 catalog_manager.cc:1347] Generated new cluster ID: 406118f705294f88b08813c9f49d0b92
I20260926 21:00:11.261153 21898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:00:11.268318 21898 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:00:11.268579 21898 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:00:11.278409 21898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6af39906e9064d3dbc3d65c38ccbfd0f: Generated new TSK 0
I20260926 21:00:11.278667 21898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:00:11.283432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 21811
2026-09-26T21:00:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:00:11 dist-test-slave-fl7g krb5kdc[21910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:00:11 dist-test-slave-fl7g krb5kdc[21910](info): set up 2 sockets
Sep 26 21:00:11 dist-test-slave-fl7g krb5kdc[21910](info): commencing operation
krb5kdc: starting...
W20260926 21:00:12.320664 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 21:00:12 dist-test-slave-fl7g krb5kdc[21910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:00:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:00:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 21:00:12.447782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37245
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:00:12.531674 21923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:12.531976 21923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:12.532021 21923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:12.533316 21923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:12.533408 21923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:12.533434 21923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:12.533457 21923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:12.533485 21923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:12.535096 21923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37245
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:12.535406 21923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:12.535658 21923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:12.540805 21931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:12.540866 21929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:12.540814 21928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:12.540913 21923 server_base.cc:1034] running on GCE node
I20260926 21:00:12.541296 21923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:12.541610 21923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:12.542815 21923 hybrid_clock.cc:648] HybridClock initialized: now 1790456412542797 us; error 28 us; skew 500 ppm
Sep 26 21:00:12 dist-test-slave-fl7g krb5kdc[21910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:12.549047 21923 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 21:00:12.549568 21923 webserver.cc:460] Webserver started at http://127.0.98.62:33373/ using document root <none> and password file <none>
I20260926 21:00:12.549763 21923 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:12.549806 21923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:12.549898 21923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:12.550969 21923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "8d72dec2e9b64316b08647d4bce8bcd3"
format_stamp: "Formatted at 2026-09-26 21:00:12 on dist-test-slave-fl7g"
server_key: "17925eda4f01175eaed375c165c09460"
server_key_iv: "779444b68461e49926e652bc25b49f35"
server_key_version: "encryptionkey@0"
I20260926 21:00:12.551290 21923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d72dec2e9b64316b08647d4bce8bcd3"
format_stamp: "Formatted at 2026-09-26 21:00:12 on dist-test-slave-fl7g"
server_key: "17925eda4f01175eaed375c165c09460"
server_key_iv: "779444b68461e49926e652bc25b49f35"
server_key_version: "encryptionkey@0"
I20260926 21:00:12.552876 21923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 21:00:12.553822 21938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:12.554064 21923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 21:00:12.554177 21923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "8d72dec2e9b64316b08647d4bce8bcd3"
format_stamp: "Formatted at 2026-09-26 21:00:12 on dist-test-slave-fl7g"
server_key: "17925eda4f01175eaed375c165c09460"
server_key_iv: "779444b68461e49926e652bc25b49f35"
server_key_version: "encryptionkey@0"
I20260926 21:00:12.554240 21923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:12.562803 21923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:12.565193 21923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:12.565306 21923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:12.569633 21923 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37245
I20260926 21:00:12.569701 21990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37245 every 8 connection(s)
I20260926 21:00:12.570056 21923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 21:00:12.570935 21991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:12.573570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 21923
I20260926 21:00:12.573681 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 21:00:12.573902 392 external_mini_cluster.cc:1428] Setting key 3db874f0652b3d7484f95feb4feabe4a
I20260926 21:00:12.574015 21991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3: Bootstrap starting.
I20260926 21:00:12.574755 21991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:12.575062 21991 log.cc:824] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:12.575896 21991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3: No bootstrap required, opened a new log
I20260926 21:00:12.578114 21991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d72dec2e9b64316b08647d4bce8bcd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37245 } }
I20260926 21:00:12.578312 21991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:00:12.578346 21991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d72dec2e9b64316b08647d4bce8bcd3, State: Initialized, Role: FOLLOWER
I20260926 21:00:12.578480 21991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d72dec2e9b64316b08647d4bce8bcd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37245 } }
I20260926 21:00:12.578570 21991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:00:12.578596 21991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:00:12.578673 21991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:12.579381 21991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d72dec2e9b64316b08647d4bce8bcd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37245 } }
I20260926 21:00:12.579514 21991 leader_election.cc:302] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d72dec2e9b64316b08647d4bce8bcd3; no voters:
I20260926 21:00:12.579663 21991 leader_election.cc:288] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:00:12.579813 21997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:00:12.579929 21991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:12.580084 21997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3 [term 1 LEADER]: Becoming Leader. State: Replica: 8d72dec2e9b64316b08647d4bce8bcd3, State: Running, Role: LEADER
I20260926 21:00:12.580185 21997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d72dec2e9b64316b08647d4bce8bcd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37245 } }
I20260926 21:00:12.580863 21996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d72dec2e9b64316b08647d4bce8bcd3. Latest consensus state: current_term: 1 leader_uuid: "8d72dec2e9b64316b08647d4bce8bcd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d72dec2e9b64316b08647d4bce8bcd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37245 } } }
I20260926 21:00:12.581018 21996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3 [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:12.581804 21998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d72dec2e9b64316b08647d4bce8bcd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d72dec2e9b64316b08647d4bce8bcd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37245 } } }
I20260926 21:00:12.581903 21998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3 [sys.catalog]: This master's current role is: LEADER
W20260926 21:00:12.582125 22012 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:00:12.582203 22012 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:00:12.582268 22005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:00:12.582439 22005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 21:00:12 dist-test-slave-fl7g krb5kdc[21910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 21:00:12.583982 22005 catalog_manager.cc:1347] Generated new cluster ID: 986d4133b70b4d9081cf2f3f1a8e025e
I20260926 21:00:12.584030 22005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:00:12.586745 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:12.574426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53139 (local address 127.0.98.62:37245)
0926 21:00:12.574629 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:12.574639 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:12.574677 (+ 38us) server_negotiation.cc:407] Connection header received
0926 21:00:12.575283 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:12.575294 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:12.575362 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:12.575522 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:12.576444 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:12.577352 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:12.578514 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:12.579084 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:12.579174 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:12.583126 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:12.583131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:12.583131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:12.583151 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:12.585083 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:12.585507 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:12.585510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:12.585511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:12.585609 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:12.585856 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:12.585857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:12.585858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:12.586118 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:12.586327 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:12.586423 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 21:00:12.586530 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
I20260926 21:00:12.587831 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:00:12.591938 22005 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:00:12.592710 22005 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:00:12.598592 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:12.589488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59609 (local address 127.0.98.62:37245)
0926 21:00:12.589531 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:12.589533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:12.591490 (+ 1957us) server_negotiation.cc:407] Connection header received
0926 21:00:12.591527 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:12.591529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:12.591545 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:12.591621 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:12.592465 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:12.593099 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:12.593882 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:12.594125 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:12.594217 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:12.595601 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:12.595603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:12.595603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:12.595612 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:12.597224 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:12.597668 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:12.597669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:12.597670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:12.597739 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:12.597999 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:12.598000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:12.598001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:12.598135 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:12.598262 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:12.598405 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 21:00:12.598477 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 21:00:12.600630 22005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d72dec2e9b64316b08647d4bce8bcd3: Generated new TSK 0
I20260926 21:00:12.600843 22005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:00:12.607060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 21923
W20260926 21:00:12.611544 21908 connection.cc:535] client connection to 127.0.98.62:37245 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:37245): Connection reset by peer (error 104)
2026-09-26T21:00:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:00:12 dist-test-slave-fl7g krb5kdc[22022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:00:12 dist-test-slave-fl7g krb5kdc[22022](info): set up 2 sockets
Sep 26 21:00:12 dist-test-slave-fl7g krb5kdc[22022](info): commencing operation
krb5kdc: starting...
W20260926 21:00:13.644649 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 21:00:13 dist-test-slave-fl7g krb5kdc[22022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:00:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:00:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 21:00:13.772686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39027
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:00:13.857251 22035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:13.857491 22035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:13.857520 22035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:13.858676 22035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:13.858721 22035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:13.858737 22035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:13.858752 22035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:13.858767 22035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:13.860267 22035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39027
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:13.860530 22035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:13.860863 22035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:13.866057 22043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:13.866124 22040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:13.866133 22041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:13.866714 22035 server_base.cc:1034] running on GCE node
I20260926 21:00:13.866959 22035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:13.867312 22035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:13.868531 22035 hybrid_clock.cc:648] HybridClock initialized: now 1790456413868507 us; error 32 us; skew 500 ppm
Sep 26 21:00:13 dist-test-slave-fl7g krb5kdc[22022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:13.871351 22035 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 21:00:13.871845 22035 webserver.cc:460] Webserver started at http://127.0.98.62:33327/ using document root <none> and password file <none>
I20260926 21:00:13.872103 22035 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:13.872197 22035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:13.872350 22035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:13.873483 22035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "74fc3b535f054ef2adfe2f3cc5d84ca6"
format_stamp: "Formatted at 2026-09-26 21:00:13 on dist-test-slave-fl7g"
server_key: "15cf8030442b603312078f5c5e7e32fc"
server_key_iv: "7ae21800b74168d4489a4df3844c1c2c"
server_key_version: "encryptionkey@0"
I20260926 21:00:13.873862 22035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "74fc3b535f054ef2adfe2f3cc5d84ca6"
format_stamp: "Formatted at 2026-09-26 21:00:13 on dist-test-slave-fl7g"
server_key: "15cf8030442b603312078f5c5e7e32fc"
server_key_iv: "7ae21800b74168d4489a4df3844c1c2c"
server_key_version: "encryptionkey@0"
I20260926 21:00:13.875345 22035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 21:00:13.876384 22050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:13.876570 22035 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:00:13.876672 22035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "74fc3b535f054ef2adfe2f3cc5d84ca6"
format_stamp: "Formatted at 2026-09-26 21:00:13 on dist-test-slave-fl7g"
server_key: "15cf8030442b603312078f5c5e7e32fc"
server_key_iv: "7ae21800b74168d4489a4df3844c1c2c"
server_key_version: "encryptionkey@0"
I20260926 21:00:13.876802 22035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:13.887288 22035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:13.889930 22035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:13.890086 22035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:13.894439 22035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39027
I20260926 21:00:13.894474 22102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39027 every 8 connection(s)
I20260926 21:00:13.894873 22035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 21:00:13.895923 22103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:13.898006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 22035
I20260926 21:00:13.898164 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 21:00:13.898357 392 external_mini_cluster.cc:1428] Setting key 3fe5aa1a6e014a19382da576745418d6
I20260926 21:00:13.898559 22103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6: Bootstrap starting.
I20260926 21:00:13.899384 22103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:13.899700 22103 log.cc:824] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:13.900501 22103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6: No bootstrap required, opened a new log
I20260926 21:00:13.902347 22103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fc3b535f054ef2adfe2f3cc5d84ca6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39027 } }
I20260926 21:00:13.902530 22103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:00:13.902578 22103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fc3b535f054ef2adfe2f3cc5d84ca6, State: Initialized, Role: FOLLOWER
I20260926 21:00:13.902757 22103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fc3b535f054ef2adfe2f3cc5d84ca6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39027 } }
I20260926 21:00:13.902863 22103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:00:13.902900 22103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:00:13.902957 22103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:13.903707 22103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fc3b535f054ef2adfe2f3cc5d84ca6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39027 } }
I20260926 21:00:13.903852 22103 leader_election.cc:302] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74fc3b535f054ef2adfe2f3cc5d84ca6; no voters:
I20260926 21:00:13.904083 22103 leader_election.cc:288] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:00:13.904120 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:13.898943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59261 (local address 127.0.98.62:39027)
0926 21:00:13.899172 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:13.899181 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:13.899207 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:00:13.899726 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:13.899733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:13.899812 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:13.899979 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:13.900233 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:13.901261 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:13.902107 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:13.902577 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:13.902726 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:13.903454 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:13.903458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:13.903458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:13.903631 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:13.903665 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:13.903676 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:13.903742 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:13.903901 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 21:00:13.903933 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":104,"threads_started":1}
I20260926 21:00:13.904362 22103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:13.904130 22108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:00:13.904691 22108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 74fc3b535f054ef2adfe2f3cc5d84ca6, State: Running, Role: LEADER
I20260926 21:00:13.904815 22108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fc3b535f054ef2adfe2f3cc5d84ca6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39027 } }
I20260926 21:00:13.905159 22109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74fc3b535f054ef2adfe2f3cc5d84ca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fc3b535f054ef2adfe2f3cc5d84ca6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39027 } } }
I20260926 21:00:13.905153 22108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74fc3b535f054ef2adfe2f3cc5d84ca6. Latest consensus state: current_term: 1 leader_uuid: "74fc3b535f054ef2adfe2f3cc5d84ca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fc3b535f054ef2adfe2f3cc5d84ca6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39027 } } }
I20260926 21:00:13.905246 22109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6 [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:13.905251 22108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6 [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:13.905275 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:00:13.905439 22117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:00:13.905987 22117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:00:13.907744 22117 catalog_manager.cc:1347] Generated new cluster ID: 00d4054096f9495bbb0ec14d62995cbf
I20260926 21:00:13.907847 22117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:00:13.910357 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:13.906884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46093 (local address 127.0.98.62:39027)
0926 21:00:13.906922 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:13.906923 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:13.907237 (+ 314us) server_negotiation.cc:407] Connection header received
0926 21:00:13.907501 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:13.907502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:13.907512 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:13.907576 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:13.907787 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:13.908280 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:13.909063 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:13.909293 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:13.909381 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:13.910100 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:13.910101 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:13.910102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:13.910106 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:13.910122 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:13.910126 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:13.910179 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:13.910249 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 21:00:13.910258 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 21:00:13.926779 22117 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:00:13.927582 22117 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:00:13.933439 22117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74fc3b535f054ef2adfe2f3cc5d84ca6: Generated new TSK 0
I20260926 21:00:13.933698 22117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:00:13.940343 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 22035
W20260926 21:00:13.944324 22020 connection.cc:535] client connection to 127.0.98.62:39027 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39027): Connection reset by peer (error 104)
2026-09-26T21:00:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:00:13 dist-test-slave-fl7g krb5kdc[22134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:00:13 dist-test-slave-fl7g krb5kdc[22134](info): set up 2 sockets
Sep 26 21:00:13 dist-test-slave-fl7g krb5kdc[22134](info): commencing operation
krb5kdc: starting...
W20260926 21:00:14.992662 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 21:00:15 dist-test-slave-fl7g krb5kdc[22134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:00:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:00:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 21:00:15.119736 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41855
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:00:15.203719 22147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:15.204010 22147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:15.204049 22147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:15.205315 22147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:15.205408 22147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:15.205435 22147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:15.205463 22147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:15.205495 22147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:15.207108 22147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41855
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:15.207441 22147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:15.207695 22147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:15.212814 22152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:15.212838 22153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:15.212982 22155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:15.213218 22147 server_base.cc:1034] running on GCE node
I20260926 21:00:15.213440 22147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:15.213794 22147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:15.214978 22147 hybrid_clock.cc:648] HybridClock initialized: now 1790456415214944 us; error 38 us; skew 500 ppm
Sep 26 21:00:15 dist-test-slave-fl7g krb5kdc[22134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:15.217653 22147 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 21:00:15.218166 22147 webserver.cc:460] Webserver started at http://127.0.98.62:43175/ using document root <none> and password file <none>
I20260926 21:00:15.218431 22147 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:15.218501 22147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:15.218653 22147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:15.219746 22147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "b43799bbd6f047c2a8cb971d0d6d5a3a"
format_stamp: "Formatted at 2026-09-26 21:00:15 on dist-test-slave-fl7g"
server_key: "c6e40c3bea932e7f4ad3c04b8a679b19"
server_key_iv: "2f4122fa9a0517fc6d4855b8ab6e9164"
server_key_version: "encryptionkey@0"
I20260926 21:00:15.220118 22147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "b43799bbd6f047c2a8cb971d0d6d5a3a"
format_stamp: "Formatted at 2026-09-26 21:00:15 on dist-test-slave-fl7g"
server_key: "c6e40c3bea932e7f4ad3c04b8a679b19"
server_key_iv: "2f4122fa9a0517fc6d4855b8ab6e9164"
server_key_version: "encryptionkey@0"
I20260926 21:00:15.221616 22147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 21:00:15.222476 22162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:15.222703 22147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:00:15.222839 22147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "b43799bbd6f047c2a8cb971d0d6d5a3a"
format_stamp: "Formatted at 2026-09-26 21:00:15 on dist-test-slave-fl7g"
server_key: "c6e40c3bea932e7f4ad3c04b8a679b19"
server_key_iv: "2f4122fa9a0517fc6d4855b8ab6e9164"
server_key_version: "encryptionkey@0"
I20260926 21:00:15.222921 22147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:15.226397 22147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:15.228802 22147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:15.228967 22147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:15.233256 22147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41855
I20260926 21:00:15.233296 22214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41855 every 8 connection(s)
I20260926 21:00:15.233644 22147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 21:00:15.234664 22215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:15.234778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 22147
I20260926 21:00:15.234896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 21:00:15.235090 392 external_mini_cluster.cc:1428] Setting key ecce2611c0b9045560f9ea61a04db133
I20260926 21:00:15.237737 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:15.235620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43945 (local address 127.0.98.62:41855)
0926 21:00:15.235816 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:15.235834 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:15.235869 (+ 35us) server_negotiation.cc:407] Connection header received
0926 21:00:15.236362 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:15.236385 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:15.236446 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:15.236597 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:15.237302 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:15.237305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:15.237305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:15.237310 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:15.237355 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:15.237369 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:15.237445 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:15.237510 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 21:00:15.237550 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260926 21:00:15.238466 22215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a: Bootstrap starting.
I20260926 21:00:15.238974 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:00:15.239176 22215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:15.239470 22215 log.cc:824] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:15.240293 22215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a: No bootstrap required, opened a new log
I20260926 21:00:15.241676 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:15.240647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60789 (local address 127.0.98.62:41855)
0926 21:00:15.240691 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:15.240693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:15.240786 (+ 93us) server_negotiation.cc:407] Connection header received
0926 21:00:15.240860 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:15.240861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:15.240864 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:15.240908 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:15.241446 (+ 538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:15.241447 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:15.241448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:15.241451 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:15.241463 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:15.241467 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:15.241505 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:15.241558 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 21:00:15.241569 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 21:00:15.242995 22215 raft_consensus.cc:348] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43799bbd6f047c2a8cb971d0d6d5a3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41855 } }
I20260926 21:00:15.243144 22215 raft_consensus.cc:374] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:00:15.243167 22215 raft_consensus.cc:729] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b43799bbd6f047c2a8cb971d0d6d5a3a, State: Initialized, Role: FOLLOWER
I20260926 21:00:15.243305 22215 consensus_queue.cc:260] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43799bbd6f047c2a8cb971d0d6d5a3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41855 } }
I20260926 21:00:15.243399 22215 raft_consensus.cc:388] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:00:15.243443 22215 raft_consensus.cc:482] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:00:15.243496 22215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:15.244102 22215 raft_consensus.cc:504] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43799bbd6f047c2a8cb971d0d6d5a3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41855 } }
I20260926 21:00:15.244187 22215 leader_election.cc:302] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b43799bbd6f047c2a8cb971d0d6d5a3a; no voters:
I20260926 21:00:15.244386 22215 leader_election.cc:288] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:00:15.244568 22225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:00:15.244668 22215 sys_catalog.cc:564] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:15.244768 22225 raft_consensus.cc:686] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a [term 1 LEADER]: Becoming Leader. State: Replica: b43799bbd6f047c2a8cb971d0d6d5a3a, State: Running, Role: LEADER
I20260926 21:00:15.244904 22225 consensus_queue.cc:237] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43799bbd6f047c2a8cb971d0d6d5a3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41855 } }
I20260926 21:00:15.245281 22227 sys_catalog.cc:455] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b43799bbd6f047c2a8cb971d0d6d5a3a. Latest consensus state: current_term: 1 leader_uuid: "b43799bbd6f047c2a8cb971d0d6d5a3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43799bbd6f047c2a8cb971d0d6d5a3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41855 } } }
I20260926 21:00:15.245299 22226 sys_catalog.cc:455] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b43799bbd6f047c2a8cb971d0d6d5a3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43799bbd6f047c2a8cb971d0d6d5a3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41855 } } }
I20260926 21:00:15.245383 22227 sys_catalog.cc:458] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:15.245414 22226 sys_catalog.cc:458] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a [sys.catalog]: This master's current role is: LEADER
W20260926 21:00:15.246466 22241 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:00:15.246511 22241 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:00:15.246572 22234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:00:15.246743 22234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:00:15.248301 22234 catalog_manager.cc:1347] Generated new cluster ID: 229e8525c8ab42b5b3c58c4f6e10d43a
I20260926 21:00:15.248359 22234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:00:15.255249 22234 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:00:15.255453 22234 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:00:15.259742 22234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b43799bbd6f047c2a8cb971d0d6d5a3a: Generated new TSK 0
I20260926 21:00:15.259969 22234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:00:15.266462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 22147
2026-09-26T21:00:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:00:15 dist-test-slave-fl7g krb5kdc[22246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:00:15 dist-test-slave-fl7g krb5kdc[22246](info): set up 2 sockets
Sep 26 21:00:15 dist-test-slave-fl7g krb5kdc[22246](info): commencing operation
krb5kdc: starting...
W20260926 21:00:16.348673 392 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 21:00:16 dist-test-slave-fl7g krb5kdc[22246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:00:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:00:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 21:00:16.476171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43165
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:00:16.558377 22259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:16.558604 22259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:16.558635 22259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:16.559759 22259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:16.559794 22259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:16.559808 22259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:16.559820 22259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:16.559834 22259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:16.561241 22259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:16.561448 22259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:16.561627 22259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:16.566694 22264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:16.566828 22259 server_base.cc:1034] running on GCE node
W20260926 21:00:16.566771 22267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:16.566730 22265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:16.567185 22259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:16.567500 22259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:16.568674 22259 hybrid_clock.cc:648] HybridClock initialized: now 1790456416568671 us; error 53 us; skew 500 ppm
Sep 26 21:00:16 dist-test-slave-fl7g krb5kdc[22246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:16.571395 22259 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 21:00:16.571941 22259 webserver.cc:460] Webserver started at http://127.0.98.62:44441/ using document root <none> and password file <none>
I20260926 21:00:16.572177 22259 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:16.572270 22259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:16.572479 22259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:16.573594 22259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "457aa1ee674d4b7a9b1d1093bdc71f16"
format_stamp: "Formatted at 2026-09-26 21:00:16 on dist-test-slave-fl7g"
server_key: "fba0beb4a77e491d7b0e8ebec4dc1885"
server_key_iv: "62bb094cec36bec614b1c94b7d66ac02"
server_key_version: "encryptionkey@0"
I20260926 21:00:16.573974 22259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "457aa1ee674d4b7a9b1d1093bdc71f16"
format_stamp: "Formatted at 2026-09-26 21:00:16 on dist-test-slave-fl7g"
server_key: "fba0beb4a77e491d7b0e8ebec4dc1885"
server_key_iv: "62bb094cec36bec614b1c94b7d66ac02"
server_key_version: "encryptionkey@0"
I20260926 21:00:16.575348 22259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 21:00:16.576187 22274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:16.576490 22259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 21:00:16.576599 22259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "457aa1ee674d4b7a9b1d1093bdc71f16"
format_stamp: "Formatted at 2026-09-26 21:00:16 on dist-test-slave-fl7g"
server_key: "fba0beb4a77e491d7b0e8ebec4dc1885"
server_key_iv: "62bb094cec36bec614b1c94b7d66ac02"
server_key_version: "encryptionkey@0"
I20260926 21:00:16.576685 22259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:16.607519 22259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:16.610457 22259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:16.610670 22259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:16.615703 22259 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43165
I20260926 21:00:16.615759 22326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43165 every 8 connection(s)
I20260926 21:00:16.616201 22259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 21:00:16.617204 22327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:16.619637 22327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16: Bootstrap starting.
I20260926 21:00:16.620292 22327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:16.620633 22327 log.cc:824] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:16.621486 22327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16: No bootstrap required, opened a new log
I20260926 21:00:16.621767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 22259
I20260926 21:00:16.621891 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 21:00:16.622098 392 external_mini_cluster.cc:1428] Setting key d18a949e8d5463375124a494eef632af
I20260926 21:00:16.624334 22327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457aa1ee674d4b7a9b1d1093bdc71f16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43165 } }
I20260926 21:00:16.624583 22327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:00:16.624689 22327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 457aa1ee674d4b7a9b1d1093bdc71f16, State: Initialized, Role: FOLLOWER
I20260926 21:00:16.624881 22327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457aa1ee674d4b7a9b1d1093bdc71f16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43165 } }
I20260926 21:00:16.624975 22327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:00:16.625012 22327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:00:16.625065 22327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:16.626168 22327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457aa1ee674d4b7a9b1d1093bdc71f16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43165 } }
I20260926 21:00:16.626276 22327 leader_election.cc:302] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 457aa1ee674d4b7a9b1d1093bdc71f16; no voters:
I20260926 21:00:16.626490 22327 leader_election.cc:288] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:00:16.626519 22332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:00:16.626633 22332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16 [term 1 LEADER]: Becoming Leader. State: Replica: 457aa1ee674d4b7a9b1d1093bdc71f16, State: Running, Role: LEADER
I20260926 21:00:16.626796 22327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:16.626785 22332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457aa1ee674d4b7a9b1d1093bdc71f16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43165 } }
I20260926 21:00:16.627288 22334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 457aa1ee674d4b7a9b1d1093bdc71f16. Latest consensus state: current_term: 1 leader_uuid: "457aa1ee674d4b7a9b1d1093bdc71f16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457aa1ee674d4b7a9b1d1093bdc71f16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43165 } } }
I20260926 21:00:16.627382 22334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16 [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:16.627588 22333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "457aa1ee674d4b7a9b1d1093bdc71f16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457aa1ee674d4b7a9b1d1093bdc71f16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43165 } } }
I20260926 21:00:16.627933 22333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16 [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:16.628136 22339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:00:16.628784 22339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 21:00:16 dist-test-slave-fl7g krb5kdc[22246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 21:00:16.631086 22339 catalog_manager.cc:1347] Generated new cluster ID: 01e4c54601d7440bb790d1474ac46c23
I20260926 21:00:16.631187 22339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:00:16.636085 22331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:16.622681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48369 (local address 127.0.98.62:43165)
0926 21:00:16.622902 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:16.622910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:16.622937 (+ 27us) server_negotiation.cc:407] Connection header received
0926 21:00:16.624342 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:16.624351 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:16.624497 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:16.624687 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:16.625542 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:16.626708 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:16.627882 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:16.628545 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:16.628693 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:16.631535 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:16.631539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:16.631539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:16.631553 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:16.633825 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:16.634264 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:16.634268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:16.634269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:16.634429 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:16.634678 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:16.634679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:16.634680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:16.635069 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:16.635262 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:16.635413 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 21:00:16.635588 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":112,"threads_started":1}
I20260926 21:00:16.637226 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:00:16.642318 22339 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:00:16.643152 22339 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:00:16.648367 22331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:16.640813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55673 (local address 127.0.98.62:43165)
0926 21:00:16.640871 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:16.640874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:16.641703 (+ 829us) server_negotiation.cc:407] Connection header received
0926 21:00:16.641816 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:16.641817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:16.641829 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:16.641863 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:16.642724 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:16.643261 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:16.643999 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:16.644229 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:16.644322 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:16.645644 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:16.645646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:16.645646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:16.645654 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:16.647152 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:16.647525 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:16.647526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:16.647527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:16.647588 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:16.647791 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:16.647792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:16.647793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:16.647913 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:16.648047 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:16.648154 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 21:00:16.648258 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 21:00:16.650422 22339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 457aa1ee674d4b7a9b1d1093bdc71f16: Generated new TSK 0
I20260926 21:00:16.650626 22339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:00:16.654520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 22259
2026-09-26T21:00:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:00:16 dist-test-slave-fl7g krb5kdc[22358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:00:16 dist-test-slave-fl7g krb5kdc[22358](info): set up 2 sockets
Sep 26 21:00:16 dist-test-slave-fl7g krb5kdc[22358](info): commencing operation
krb5kdc: starting...
W20260926 21:00:17.716951 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 21:00:17 dist-test-slave-fl7g krb5kdc[22358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:00:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:00:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 21:00:17.845117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43805
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:00:17.929405 22371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:17.929687 22371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:17.929733 22371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:17.930979 22371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:17.931062 22371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:17.931088 22371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:17.931120 22371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:17.931162 22371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:17.932817 22371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:17.933100 22371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:17.933346 22371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:17.938550 22376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:17.938608 22377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:17.938604 22379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:17.938839 22371 server_base.cc:1034] running on GCE node
I20260926 21:00:17.939121 22371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:17.939431 22371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:17.940604 22371 hybrid_clock.cc:648] HybridClock initialized: now 1790456417940591 us; error 60 us; skew 500 ppm
Sep 26 21:00:17 dist-test-slave-fl7g krb5kdc[22358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:17.950466 22371 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 21:00:17.950980 22371 webserver.cc:460] Webserver started at http://127.0.98.62:38963/ using document root <none> and password file <none>
I20260926 21:00:17.951196 22371 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:17.951241 22371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:17.951339 22371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:17.952458 22371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "3b7788acfa0949c18644bfa48ea0c2d4"
format_stamp: "Formatted at 2026-09-26 21:00:17 on dist-test-slave-fl7g"
server_key: "f779c2716ac2be2c9fc12078c83014ed"
server_key_iv: "7ff67acb41dd0ee8ddc8e50b29138603"
server_key_version: "encryptionkey@0"
I20260926 21:00:17.952829 22371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b7788acfa0949c18644bfa48ea0c2d4"
format_stamp: "Formatted at 2026-09-26 21:00:17 on dist-test-slave-fl7g"
server_key: "f779c2716ac2be2c9fc12078c83014ed"
server_key_iv: "7ff67acb41dd0ee8ddc8e50b29138603"
server_key_version: "encryptionkey@0"
I20260926 21:00:17.954181 22371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:00:17.954969 22386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:17.955193 22371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:00:17.955287 22371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "3b7788acfa0949c18644bfa48ea0c2d4"
format_stamp: "Formatted at 2026-09-26 21:00:17 on dist-test-slave-fl7g"
server_key: "f779c2716ac2be2c9fc12078c83014ed"
server_key_iv: "7ff67acb41dd0ee8ddc8e50b29138603"
server_key_version: "encryptionkey@0"
I20260926 21:00:17.955346 22371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:17.971288 22371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:17.973836 22371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:17.974015 22371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:17.978031 22371 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43805
I20260926 21:00:17.978078 22438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43805 every 8 connection(s)
I20260926 21:00:17.978440 22371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 21:00:17.979336 22439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:17.980376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 22371
I20260926 21:00:17.980540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 21:00:17.980731 392 external_mini_cluster.cc:1428] Setting key dd53e85b40e89406b5eb0a52e21a3ec7
I20260926 21:00:17.982076 22439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4: Bootstrap starting.
I20260926 21:00:17.982678 22439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:17.982967 22439 log.cc:824] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:17.983672 22439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4: No bootstrap required, opened a new log
I20260926 21:00:17.985644 22439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7788acfa0949c18644bfa48ea0c2d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43805 } }
I20260926 21:00:17.985831 22439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:00:17.985885 22439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b7788acfa0949c18644bfa48ea0c2d4, State: Initialized, Role: FOLLOWER
I20260926 21:00:17.986065 22439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7788acfa0949c18644bfa48ea0c2d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43805 } }
I20260926 21:00:17.986148 22439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:00:17.986171 22439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:00:17.986287 22439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:17.986621 22442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:17.981244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44289 (local address 127.0.98.62:43805)
0926 21:00:17.981437 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:17.981447 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:17.981479 (+ 32us) server_negotiation.cc:407] Connection header received
0926 21:00:17.981955 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:17.981964 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:17.982030 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:17.982197 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:17.982431 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:17.983562 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:17.984505 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:17.984955 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:17.985093 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:17.985881 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:17.985888 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:17.985888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:17.986103 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:17.986139 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:17.986172 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:17.986232 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:17.986313 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 21:00:17.986395 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260926 21:00:17.987039 22439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7788acfa0949c18644bfa48ea0c2d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43805 } }
I20260926 21:00:17.987201 22439 leader_election.cc:302] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b7788acfa0949c18644bfa48ea0c2d4; no voters:
I20260926 21:00:17.987401 22439 leader_election.cc:288] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:00:17.987448 22444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:00:17.987613 22444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4 [term 1 LEADER]: Becoming Leader. State: Replica: 3b7788acfa0949c18644bfa48ea0c2d4, State: Running, Role: LEADER
I20260926 21:00:17.987679 22439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:17.987759 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:00:17.987758 22444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7788acfa0949c18644bfa48ea0c2d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43805 } }
I20260926 21:00:17.988046 22444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b7788acfa0949c18644bfa48ea0c2d4. Latest consensus state: current_term: 1 leader_uuid: "3b7788acfa0949c18644bfa48ea0c2d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7788acfa0949c18644bfa48ea0c2d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43805 } } }
I20260926 21:00:17.988057 22446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b7788acfa0949c18644bfa48ea0c2d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7788acfa0949c18644bfa48ea0c2d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43805 } } }
I20260926 21:00:17.988164 22444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4 [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:17.988242 22446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4 [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:17.988440 22450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:00:17.989005 22450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:00:17.990664 22450 catalog_manager.cc:1347] Generated new cluster ID: d2006064e3fb440a94d6b2a3f677f0b8
I20260926 21:00:17.990712 22450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:00:17.993439 22442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:17.989755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38001 (local address 127.0.98.62:43805)
0926 21:00:17.989790 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:17.989792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:17.989918 (+ 126us) server_negotiation.cc:407] Connection header received
0926 21:00:17.989996 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:17.989997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:17.990005 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:17.990089 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:17.990276 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:17.991004 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:17.992052 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:17.992344 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:17.992386 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:17.993175 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:17.993176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:17.993177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:17.993181 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:17.993195 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:17.993199 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:17.993228 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:17.993336 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 21:00:17.993361 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 21:00:18.005779 22450 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:00:18.006361 22450 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:00:18.015869 22450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b7788acfa0949c18644bfa48ea0c2d4: Generated new TSK 0
I20260926 21:00:18.016127 22450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:00:18.022279 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 22371
2026-09-26T21:00:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:00:18 dist-test-slave-fl7g krb5kdc[22470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:00:18 dist-test-slave-fl7g krb5kdc[22470](info): set up 2 sockets
Sep 26 21:00:18 dist-test-slave-fl7g krb5kdc[22470](info): commencing operation
krb5kdc: starting...
W20260926 21:00:19.064656 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 21:00:19 dist-test-slave-fl7g krb5kdc[22470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:00:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:00:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 21:00:19.192005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43151
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:00:19.274922 22483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:19.275187 22483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:19.275213 22483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:19.276717 22483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:19.276784 22483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:19.276804 22483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:19.276819 22483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:19.276834 22483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:19.278609 22483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:19.278872 22483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:19.279080 22483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:19.284718 22491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:19.284749 22489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:19.284801 22488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:19.284905 22483 server_base.cc:1034] running on GCE node
I20260926 21:00:19.285346 22483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:19.285724 22483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:19.286897 22483 hybrid_clock.cc:648] HybridClock initialized: now 1790456419286878 us; error 32 us; skew 500 ppm
Sep 26 21:00:19 dist-test-slave-fl7g krb5kdc[22470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:19.289541 22483 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 21:00:19.290056 22483 webserver.cc:460] Webserver started at http://127.0.98.62:45171/ using document root <none> and password file <none>
I20260926 21:00:19.290256 22483 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:19.290310 22483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:19.290409 22483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:19.291469 22483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "cc0fe42ab5fd4bdc92218206e8ce94df"
format_stamp: "Formatted at 2026-09-26 21:00:19 on dist-test-slave-fl7g"
server_key: "f652ee9b73b8608d21648a214baaeb14"
server_key_iv: "b72cf9850ddafff78f771f525dd151c2"
server_key_version: "encryptionkey@0"
I20260926 21:00:19.291791 22483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "cc0fe42ab5fd4bdc92218206e8ce94df"
format_stamp: "Formatted at 2026-09-26 21:00:19 on dist-test-slave-fl7g"
server_key: "f652ee9b73b8608d21648a214baaeb14"
server_key_iv: "b72cf9850ddafff78f771f525dd151c2"
server_key_version: "encryptionkey@0"
I20260926 21:00:19.293273 22483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 21:00:19.294239 22498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:19.294523 22483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:00:19.294683 22483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "cc0fe42ab5fd4bdc92218206e8ce94df"
format_stamp: "Formatted at 2026-09-26 21:00:19 on dist-test-slave-fl7g"
server_key: "f652ee9b73b8608d21648a214baaeb14"
server_key_iv: "b72cf9850ddafff78f771f525dd151c2"
server_key_version: "encryptionkey@0"
I20260926 21:00:19.294778 22483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:19.298486 22483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:19.300853 22483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:19.301038 22483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:19.305596 22483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43151
I20260926 21:00:19.305637 22550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43151 every 8 connection(s)
I20260926 21:00:19.306030 22483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 21:00:19.307142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 22483
I20260926 21:00:19.307199 22551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:19.307291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 21:00:19.307492 392 external_mini_cluster.cc:1428] Setting key dc78c4b159924aa70b4ea00b6180c13e
I20260926 21:00:19.310536 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:19.308085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55419 (local address 127.0.98.62:43151)
0926 21:00:19.308299 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:19.308309 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:19.308335 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:00:19.308916 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:19.308924 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:19.308961 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:19.309121 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:19.309981 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:19.309985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:19.309985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:19.309989 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:19.310049 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:19.310060 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:19.310143 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:19.310213 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 21:00:19.310376 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":96,"threads_started":1}
I20260926 21:00:19.311134 22551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df: Bootstrap starting.
I20260926 21:00:19.311748 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:00:19.311982 22551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:19.312279 22551 log.cc:824] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:19.313197 22551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df: No bootstrap required, opened a new log
I20260926 21:00:19.314633 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:19.313466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45801 (local address 127.0.98.62:43151)
0926 21:00:19.313514 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:19.313516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:19.313577 (+ 61us) server_negotiation.cc:407] Connection header received
0926 21:00:19.313661 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:19.313662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:19.313668 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:19.313718 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:19.314392 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:19.314394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:19.314394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:00:19.314399 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:19.314422 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:00:19.314427 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:19.314483 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:19.314549 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 21:00:19.314564 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 21:00:19.315577 22551 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0fe42ab5fd4bdc92218206e8ce94df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } }
I20260926 21:00:19.315770 22551 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:00:19.315824 22551 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc0fe42ab5fd4bdc92218206e8ce94df, State: Initialized, Role: FOLLOWER
I20260926 21:00:19.315965 22551 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0fe42ab5fd4bdc92218206e8ce94df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } }
I20260926 21:00:19.316066 22551 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:00:19.316111 22551 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:00:19.316167 22551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:19.316865 22551 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0fe42ab5fd4bdc92218206e8ce94df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } }
I20260926 21:00:19.316995 22551 leader_election.cc:302] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc0fe42ab5fd4bdc92218206e8ce94df; no voters:
I20260926 21:00:19.317214 22551 leader_election.cc:288] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:00:19.317273 22561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:00:19.317466 22561 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df [term 1 LEADER]: Becoming Leader. State: Replica: cc0fe42ab5fd4bdc92218206e8ce94df, State: Running, Role: LEADER
I20260926 21:00:19.317512 22551 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:19.317628 22561 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0fe42ab5fd4bdc92218206e8ce94df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } }
I20260926 21:00:19.318059 22562 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc0fe42ab5fd4bdc92218206e8ce94df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0fe42ab5fd4bdc92218206e8ce94df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } }
I20260926 21:00:19.318070 22563 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc0fe42ab5fd4bdc92218206e8ce94df. Latest consensus state: current_term: 1 leader_uuid: "cc0fe42ab5fd4bdc92218206e8ce94df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0fe42ab5fd4bdc92218206e8ce94df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43151 } } }
I20260926 21:00:19.318197 22562 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:19.318219 22563 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:19.318459 22570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:00:19.318972 22570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:00:19.320807 22570 catalog_manager.cc:1347] Generated new cluster ID: 694831a3095547ba8e0fb4b5bc6c05dd
I20260926 21:00:19.320866 22570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:00:19.346753 22570 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:00:19.347051 22570 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:00:19.349617 22570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc0fe42ab5fd4bdc92218206e8ce94df: Generated new TSK 0
I20260926 21:00:19.349831 22570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:00:19.355495 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 22483
2026-09-26T21:00:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1335 ms)
[----------] 18 tests from AuthTokenIssuingTest (24247 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:00:19 dist-test-slave-fl7g krb5kdc[22582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:00:19 dist-test-slave-fl7g krb5kdc[22582](info): set up 2 sockets
Sep 26 21:00:19 dist-test-slave-fl7g krb5kdc[22582](info): commencing operation
krb5kdc: starting...
W20260926 21:00:20.392781 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 21:00:20 dist-test-slave-fl7g krb5kdc[22582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:00:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:00:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 21:00:20.516637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46675
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46675,127.0.98.61:40265,127.0.98.60:38843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:00:20.601984 22595 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 21:00:20.602206 22595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:20.602226 22595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:20.602247 22595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:20.603436 22595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:20.603479 22595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:20.603494 22595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:20.603507 22595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:20.603564 22595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:20.605103 22595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46675,127.0.98.61:40265,127.0.98.60:38843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:20.605376 22595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:20.605618 22595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:20.610749 22601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:20.610737 22600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:20.610740 22603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:20.611121 22595 server_base.cc:1034] running on GCE node
I20260926 21:00:20.611402 22595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:20.611737 22595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:20.612912 22595 hybrid_clock.cc:648] HybridClock initialized: now 1790456420612894 us; error 37 us; skew 500 ppm
Sep 26 21:00:20 dist-test-slave-fl7g krb5kdc[22582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:20.615787 22595 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 21:00:20.616326 22595 webserver.cc:460] Webserver started at http://127.0.98.62:40223/ using document root <none> and password file <none>
I20260926 21:00:20.616659 22595 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:20.616710 22595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:20.616849 22595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:20.617959 22595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "62c8fce264a449cca64793667863915d"
format_stamp: "Formatted at 2026-09-26 21:00:20 on dist-test-slave-fl7g"
server_key: "446be710df9d6e0b3fff697b61094acc"
server_key_iv: "0a3c6ed2bb511fd8cae25f61127ab939"
server_key_version: "encryptionkey@0"
I20260926 21:00:20.618333 22595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "62c8fce264a449cca64793667863915d"
format_stamp: "Formatted at 2026-09-26 21:00:20 on dist-test-slave-fl7g"
server_key: "446be710df9d6e0b3fff697b61094acc"
server_key_iv: "0a3c6ed2bb511fd8cae25f61127ab939"
server_key_version: "encryptionkey@0"
I20260926 21:00:20.619751 22595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 21:00:20.620616 22610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:20.620814 22595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:00:20.620911 22595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "62c8fce264a449cca64793667863915d"
format_stamp: "Formatted at 2026-09-26 21:00:20 on dist-test-slave-fl7g"
server_key: "446be710df9d6e0b3fff697b61094acc"
server_key_iv: "0a3c6ed2bb511fd8cae25f61127ab939"
server_key_version: "encryptionkey@0"
I20260926 21:00:20.620999 22595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:20.637200 22595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:20.639780 22595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:20.639986 22595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:20.644273 22595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46675
I20260926 21:00:20.644342 22662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46675 every 8 connection(s)
I20260926 21:00:20.644791 22595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 21:00:20.645866 22663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:20.647495 22663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:00:20.652243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 22595
I20260926 21:00:20.652338 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 21:00:20.652590 392 external_mini_cluster.cc:1428] Setting key 6e41cd3af5b7442115d543514b2360e6
Sep 26 21:00:20 dist-test-slave-fl7g krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 21:00:20.658257 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.648310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46732 (local address 127.0.98.62:46675)
0926 21:00:20.648496 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:20.648506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:20.648641 (+ 135us) server_negotiation.cc:407] Connection header received
0926 21:00:20.649230 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:20.649233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:20.649253 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:20.649361 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:20.649931 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.650752 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.651654 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.651981 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.652119 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.654692 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:20.654699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:20.654700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:20.654706 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:20.656552 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.656967 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.656970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.656971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:20.657058 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.657301 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.657302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.657302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:20.657539 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:20.657635 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:20.657819 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 21:00:20.657994 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":152,"thread_start_us":78,"threads_started":1}
I20260926 21:00:20.658265 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.648211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46675 (local address 127.0.0.1:46732)
0926 21:00:20.648486 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.648533 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 21:00:20.649059 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:00:20.649365 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:00:20.649368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:00:20.649655 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:00:20.649820 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:20.649824 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.650866 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.650867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:20.651552 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:20.651553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.652158 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.652159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:20.652236 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.653015 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:00:20.653025 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:00:20.654595 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:00:20.656648 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:20.656655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:20.656657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:20.656866 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:20.657127 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:20.657128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:20.657128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:20.657232 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:20.657610 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:00:20.657615 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:00:20.657680 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 21:00:20.657794 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 21:00:20.658042 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":108,"thread_start_us":130,"threads_started":1}
I20260926 21:00:20.659415 22663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:00:20.659844 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.659687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40265 (local address 127.0.0.1:47312)
0926 21:00:20.659779 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.659820 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 21:00:20.659974 22613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40265: connect: Connection refused (error 111)
I20260926 21:00:20.660467 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.660331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40265 (local address 127.0.0.1:47318)
0926 21:00:20.660374 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.660398 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 21:00:20.660578 22663 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40265: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40265: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 21:00:20.680397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40265
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46675,127.0.98.61:40265,127.0.98.60:38843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:00:20.709863 22663 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } attempt: 1
I20260926 21:00:20.710441 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.710260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40265 (local address 127.0.0.1:47322)
0926 21:00:20.710343 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.710400 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 21:00:20.710911 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.710802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40265 (local address 127.0.0.1:47330)
0926 21:00:20.710853 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.710887 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 21:00:20.711014 22663 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40265: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40265: connect: Connection refused (error 111)
W20260926 21:00:20.776319 22670 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 21:00:20.776629 22670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:20.776647 22670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:20.776669 22670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:20.777853 22670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:20.777894 22670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:20.777909 22670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:20.777921 22670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:20.777935 22670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:20.779150 22663 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } attempt: 2
I20260926 21:00:20.779598 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.779454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40265 (local address 127.0.0.1:47336)
0926 21:00:20.779520 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.779565 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 21:00:20.779659 22670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46675,127.0.98.61:40265,127.0.98.60:38843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:20.780025 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.779923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40265 (local address 127.0.0.1:47340)
0926 21:00:20.779972 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.780008 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 21:00:20.780138 22663 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40265: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40265: connect: Connection refused (error 111)
I20260926 21:00:20.780193 22670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:20.780488 22670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:20.785476 22676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:20.785476 22678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:20.785506 22675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:20.785522 22670 server_base.cc:1034] running on GCE node
I20260926 21:00:20.786010 22670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:20.786378 22670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:20.787560 22670 hybrid_clock.cc:648] HybridClock initialized: now 1790456420787554 us; error 38 us; skew 500 ppm
Sep 26 21:00:20 dist-test-slave-fl7g krb5kdc[22582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:20.790408 22670 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 21:00:20.790874 22670 webserver.cc:460] Webserver started at http://127.0.98.61:35813/ using document root <none> and password file <none>
I20260926 21:00:20.791072 22670 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:20.791117 22670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:20.791239 22670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:20.792347 22670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-1/data/instance:
uuid: "41a74c468a4b4dca8938386e9c2aa72b"
format_stamp: "Formatted at 2026-09-26 21:00:20 on dist-test-slave-fl7g"
server_key: "eecd6a10358877b8434c7cf7e39a1e73"
server_key_iv: "1a1d8d2725bd0380a43fc94da4a9452c"
server_key_version: "encryptionkey@0"
I20260926 21:00:20.792794 22670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-1/wal/instance:
uuid: "41a74c468a4b4dca8938386e9c2aa72b"
format_stamp: "Formatted at 2026-09-26 21:00:20 on dist-test-slave-fl7g"
server_key: "eecd6a10358877b8434c7cf7e39a1e73"
server_key_iv: "1a1d8d2725bd0380a43fc94da4a9452c"
server_key_version: "encryptionkey@0"
I20260926 21:00:20.794246 22670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:00:20.795080 22685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:20.795321 22670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:00:20.795437 22670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-1/wal
uuid: "41a74c468a4b4dca8938386e9c2aa72b"
format_stamp: "Formatted at 2026-09-26 21:00:20 on dist-test-slave-fl7g"
server_key: "eecd6a10358877b8434c7cf7e39a1e73"
server_key_iv: "1a1d8d2725bd0380a43fc94da4a9452c"
server_key_version: "encryptionkey@0"
I20260926 21:00:20.795516 22670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:20.807415 22670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:20.810539 22670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:20.810715 22670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:20.814854 22670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40265
I20260926 21:00:20.814908 22737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40265 every 8 connection(s)
I20260926 21:00:20.815266 22670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-1/data/info.pb
I20260926 21:00:20.815678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 22670
I20260926 21:00:20.815830 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-1/wal/instance
I20260926 21:00:20.816128 392 external_mini_cluster.cc:1428] Setting key c4e7403a1fa25d92696656ddc9b03459
I20260926 21:00:20.816634 22738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:20.818594 22738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 21:00:20 dist-test-slave-fl7g krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 21:00:20.830641 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.819451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46742 (local address 127.0.98.62:46675)
0926 21:00:20.819521 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:20.819523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:20.819711 (+ 188us) server_negotiation.cc:407] Connection header received
0926 21:00:20.820257 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:20.820259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:20.820275 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:20.820341 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:20.821085 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.821586 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.822614 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.822839 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.822919 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.827383 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:20.827385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:20.827385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:20.827395 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:20.829006 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.829564 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.829565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.829565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:20.829617 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.829918 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.829919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.829919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:20.830029 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:20.830116 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:20.830498 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 21:00:20.830570 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 21:00:20.831044 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.819373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46675 (local address 127.0.0.1:46742)
0926 21:00:20.819583 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.819622 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 21:00:20.820133 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:00:20.820497 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:00:20.820501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:00:20.820773 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:00:20.820982 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:20.820986 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.821687 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.821688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:20.822505 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:20.822507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.822941 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.822942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:20.823082 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.823719 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:00:20.823728 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:00:20.827263 (+ 3535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:00:20.829106 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:20.829115 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:20.829118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:20.829483 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:20.829685 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:20.829686 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:20.829687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:20.829851 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:20.830124 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:00:20.830133 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:00:20.830347 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 21:00:20.830479 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 21:00:20.830672 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":99,"threads_started":1}
I20260926 21:00:20.831795 22738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:00:20 dist-test-slave-fl7g krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 21:00:20.841353 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.832100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40265 (local address 127.0.0.1:47352)
0926 21:00:20.832176 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.832189 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 21:00:20.832242 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:00:20.832583 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:00:20.832584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:00:20.832735 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:00:20.832787 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:20.832788 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.833681 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.833682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:20.834233 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:20.834235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.834749 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.834750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:20.834816 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.835292 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:00:20.835298 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:00:20.837626 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:00:20.839997 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:20.839999 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:20.839999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:20.840214 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:20.840584 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:20.840585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:20.840586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:20.840669 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:20.841036 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:00:20.841037 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:00:20.841094 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 21:00:20.841183 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 21:00:20.841279 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 21:00:20.841506 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.832211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47352 (local address 127.0.98.61:40265)
0926 21:00:20.832383 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:20.832387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:20.832404 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:00:20.832464 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:20.832467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:20.832480 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:20.832580 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:20.832866 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.833583 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.834333 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.834667 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.834752 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.837738 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:20.837770 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:20.837771 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:20.837780 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:20.839887 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.840352 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.840355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.840355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:20.840454 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.840735 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.840735 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.840736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:20.840922 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:20.840987 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:20.841210 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 21:00:20.841346 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260926 21:00:20.842221 22738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:00:20.842620 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.842492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38843 (local address 127.0.0.1:46440)
0926 21:00:20.842549 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.842598 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 21:00:20.842765 22688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38843: connect: Connection refused (error 111)
I20260926 21:00:20.843262 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.843127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38843 (local address 127.0.0.1:46444)
0926 21:00:20.843203 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.843230 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 21:00:20.843366 22738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38843: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38843: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 21:00:20.844345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38843
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46675,127.0.98.61:40265,127.0.98.60:38843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:00:20.871312 22663 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } attempt: 3
Sep 26 21:00:20 dist-test-slave-fl7g krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 21:00:20.878726 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.871787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40265 (local address 127.0.0.1:47356)
0926 21:00:20.871845 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.871858 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 21:00:20.871935 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:00:20.872099 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:00:20.872100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:00:20.872288 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:00:20.872355 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:20.872357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.873070 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.873071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:20.873621 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:20.873623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.874022 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.874023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:20.874111 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.874611 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:00:20.874615 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:00:20.875848 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:00:20.877615 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:20.877616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:20.877617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:20.877802 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:20.878089 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:20.878090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:20.878090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:20.878155 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:20.878455 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:00:20.878456 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:00:20.878500 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 21:00:20.878558 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 21:00:20.878641 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 21:00:20.878789 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.871872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47356 (local address 127.0.98.61:40265)
0926 21:00:20.871949 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:20.871951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:20.871962 (+ 11us) server_negotiation.cc:407] Connection header received
0926 21:00:20.872003 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:20.872004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:20.872016 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:20.872092 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:20.872462 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.872973 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.873714 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.873946 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.874026 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.875939 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:20.875941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:20.875941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:20.875949 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:20.877510 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.877913 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.877915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.877915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:20.877998 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.878227 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.878228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.878228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:20.878385 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:20.878460 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:20.878606 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 21:00:20.878699 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 21:00:20.879307 22663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:00:20.879686 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.879570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38843 (local address 127.0.0.1:46452)
0926 21:00:20.879622 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.879657 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 21:00:20.880048 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.879970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38843 (local address 127.0.0.1:46456)
0926 21:00:20.880008 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.880032 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 21:00:20.880151 22663 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38843: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38843: connect: Connection refused (error 111)
I20260926 21:00:20.892586 22738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } attempt: 1
I20260926 21:00:20.893144 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.892984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38843 (local address 127.0.0.1:46470)
0926 21:00:20.893053 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.893106 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 21:00:20.893530 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.893438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38843 (local address 127.0.0.1:46486)
0926 21:00:20.893483 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.893511 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 21:00:20.893646 22738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38843: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38843: connect: Connection refused (error 111)
I20260926 21:00:20.911317 22663 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } attempt: 1
I20260926 21:00:20.911837 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.911659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38843 (local address 127.0.0.1:46496)
0926 21:00:20.911732 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.911788 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 21:00:20.912297 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.912189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38843 (local address 127.0.0.1:46500)
0926 21:00:20.912238 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.912269 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 21:00:20.912437 22663 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38843: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38843: connect: Connection refused (error 111)
W20260926 21:00:20.930660 22745 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 21:00:20.930903 22745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:20.930922 22745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:20.930943 22745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:20.932139 22745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:20.932189 22745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:20.932204 22745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:20.932216 22745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:20.932229 22745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:20.933966 22745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46675,127.0.98.61:40265,127.0.98.60:38843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:20.934252 22745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:20.934531 22745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:20.939545 22750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:20.939661 22751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:20.939545 22753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:20.940389 22745 server_base.cc:1034] running on GCE node
I20260926 21:00:20.940737 22745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:20.941112 22745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:20.942313 22745 hybrid_clock.cc:648] HybridClock initialized: now 1790456420942280 us; error 35 us; skew 500 ppm
Sep 26 21:00:20 dist-test-slave-fl7g krb5kdc[22582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:20.945726 22745 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 21:00:20.946211 22745 webserver.cc:460] Webserver started at http://127.0.98.60:41013/ using document root <none> and password file <none>
I20260926 21:00:20.946456 22745 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:20.946503 22745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:20.946652 22745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:20.947749 22745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-2/data/instance:
uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1"
format_stamp: "Formatted at 2026-09-26 21:00:20 on dist-test-slave-fl7g"
server_key: "1487aa9704cae651e0c36bb6b4c20aa0"
server_key_iv: "712403c13c1e86269b184a87e009f3b4"
server_key_version: "encryptionkey@0"
I20260926 21:00:20.948164 22745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-2/wal/instance:
uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1"
format_stamp: "Formatted at 2026-09-26 21:00:20 on dist-test-slave-fl7g"
server_key: "1487aa9704cae651e0c36bb6b4c20aa0"
server_key_iv: "712403c13c1e86269b184a87e009f3b4"
server_key_version: "encryptionkey@0"
I20260926 21:00:20.949631 22745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 21:00:20.950416 22760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:20.950606 22745 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:00:20.950698 22745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-2/wal
uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1"
format_stamp: "Formatted at 2026-09-26 21:00:20 on dist-test-slave-fl7g"
server_key: "1487aa9704cae651e0c36bb6b4c20aa0"
server_key_iv: "712403c13c1e86269b184a87e009f3b4"
server_key_version: "encryptionkey@0"
I20260926 21:00:20.950773 22745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:20.961505 22745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:20.961854 22738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } attempt: 2
I20260926 21:00:20.962368 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.962195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38843 (local address 127.0.0.1:46502)
0926 21:00:20.962264 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.962330 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 21:00:20.962836 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.962714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38843 (local address 127.0.0.1:46512)
0926 21:00:20.962760 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.962789 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 21:00:20.962944 22738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38843: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38843: connect: Connection refused (error 111)
I20260926 21:00:20.964102 22745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:20.964263 22745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:20.968073 22745 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38843
I20260926 21:00:20.968146 22812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38843 every 8 connection(s)
I20260926 21:00:20.968575 22745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-2/data/info.pb
I20260926 21:00:20.969468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 22745
I20260926 21:00:20.969553 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790456150796563-392-0/minicluster-data/master-2/wal/instance
I20260926 21:00:20.969733 22813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:20.969800 392 external_mini_cluster.cc:1428] Setting key 3ead80bd2ee0cc7bcae9419c9ee8208a
I20260926 21:00:20.971796 22813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:00:20 dist-test-slave-fl7g krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 21:00:20 dist-test-slave-fl7g krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 21:00:20.980902 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.972486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.0.98.62:46675)
0926 21:00:20.972540 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:20.972542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:20.972702 (+ 160us) server_negotiation.cc:407] Connection header received
0926 21:00:20.972728 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:20.972729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:20.972737 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:20.972785 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:20.973361 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.973846 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.974698 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.974891 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.974985 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.977375 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:20.977377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:20.977377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:20.977385 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:20.979119 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.979672 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.979673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.979673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:20.979740 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.980054 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.980055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.980055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:20.980213 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:20.980294 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:20.980657 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 21:00:20.980800 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 21:00:20.981030 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.972381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46675 (local address 127.0.0.1:46744)
0926 21:00:20.972571 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.972612 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 21:00:20.972681 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:00:20.972808 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:00:20.972811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:00:20.973146 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:00:20.973251 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:20.973257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.973960 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.973961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:20.974599 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:20.974602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.975005 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.975006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:20.975057 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.975436 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:00:20.975446 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:00:20.977268 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:00:20.979244 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:20.979252 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:20.979254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:20.979590 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:20.979810 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:20.979811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:20.979812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:20.979969 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:20.980292 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:00:20.980297 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:00:20.980501 (+ 204us) client_negotiation.cc:769] Sending connection context
0926 21:00:20.980622 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 21:00:20.980753 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":27,"thread_start_us":119,"threads_started":1}
I20260926 21:00:20.981664 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.970343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46522 (local address 127.0.98.60:38843)
0926 21:00:20.970540 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:20.970550 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:20.970582 (+ 32us) server_negotiation.cc:407] Connection header received
0926 21:00:20.971128 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:20.971135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:20.971184 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:20.971346 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:20.971987 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.973373 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.974128 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.974575 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.974696 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.977855 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:20.977864 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:20.977864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:20.977872 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:20.979850 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.980241 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.980244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.980244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:20.980311 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.980548 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.980549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.980549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:20.980748 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:20.980902 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:20.981438 (+ 536us) server_negotiation.cc:299] Negotiation successful
0926 21:00:20.981546 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260926 21:00:20.981657 22813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:00:20 dist-test-slave-fl7g krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 21:00:20 dist-test-slave-fl7g krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 21:00:20.987594 22663 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } attempt: 2
I20260926 21:00:20.989585 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.982000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40265 (local address 127.0.0.1:47370)
0926 21:00:20.982040 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.982053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 21:00:20.982098 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:00:20.982295 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:00:20.982296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:00:20.982444 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:00:20.982489 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:20.982491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.983326 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.983327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:20.983882 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:20.983883 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.984313 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.984314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:20.984398 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.984944 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:00:20.984949 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:00:20.986429 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:00:20.988252 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:20.988254 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:20.988254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:20.988549 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:20.988811 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:20.988811 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:20.988812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:20.988882 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:20.989187 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:00:20.989188 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:00:20.989239 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 21:00:20.989368 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 21:00:20.989474 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 21:00:20.989591 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.982074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.0.98.61:40265)
0926 21:00:20.982153 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:20.982156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:20.982165 (+ 9us) server_negotiation.cc:407] Connection header received
0926 21:00:20.982193 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:20.982194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:20.982204 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:20.982283 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:20.982573 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.983233 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.983972 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.984230 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.984318 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.986521 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:20.986522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:20.986522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:20.986530 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:20.988171 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.988658 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.988660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.988660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:20.988726 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.988959 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.988960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.988960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:20.989111 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:20.989187 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:20.989389 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 21:00:20.989468 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 21:00:20.990525 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.982754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.0.98.62:46675)
0926 21:00:20.982797 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:20.982799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:20.982807 (+ 8us) server_negotiation.cc:407] Connection header received
0926 21:00:20.982827 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:20.982828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:20.982836 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:20.982902 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:20.983343 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.983803 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.984565 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.984770 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.984866 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.987183 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:20.987185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:20.987185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:20.987192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:20.989236 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.989608 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.989609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.989610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:20.989667 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.989906 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.989908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.989908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:20.990065 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:20.990261 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:20.990339 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 21:00:20.990425 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 21:00:20.990552 22813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:00:20 dist-test-slave-fl7g krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 21:00:20 dist-test-slave-fl7g krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 21:00:20.996855 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.987899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38843 (local address 127.0.0.1:46530)
0926 21:00:20.987945 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.987957 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 21:00:20.987997 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:00:20.988600 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:00:20.988601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:00:20.988923 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:00:20.988977 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:20.988980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.989974 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.989975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:20.991009 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:20.991011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.991539 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.991540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:20.991600 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.991999 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:00:20.992004 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:00:20.993399 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:00:20.995457 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:20.995458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:20.995459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:20.995767 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:20.996033 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:20.996034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:20.996034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:20.996110 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:20.996516 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:00:20.996517 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:00:20.996576 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 21:00:20.996633 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 21:00:20.996743 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 21:00:20.997278 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.988071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.0.98.60:38843)
0926 21:00:20.988450 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:20.988452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:20.988461 (+ 9us) server_negotiation.cc:407] Connection header received
0926 21:00:20.988482 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:20.988483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:20.988495 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:20.988805 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:20.989059 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.989893 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.991063 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.991430 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.991545 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.993491 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:20.993492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:20.993492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:20.993498 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:20.995362 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.995869 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.995870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.995871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:20.995925 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.996188 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.996189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.996190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:20.996366 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:20.997063 (+ 697us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:20.997109 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 21:00:20.997189 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326}
Sep 26 21:00:20 dist-test-slave-fl7g krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 21:00:20.999456 22663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d: Bootstrap starting.
I20260926 21:00:21.000171 22663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:21.000566 22663 log.cc:824] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:21.000782 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.992004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47372 (local address 127.0.98.61:40265)
0926 21:00:20.992045 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:20.992047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:20.992055 (+ 8us) server_negotiation.cc:407] Connection header received
0926 21:00:20.992076 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:20.992077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:20.992086 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:20.992147 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:20.992824 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.993543 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.994745 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.995180 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.995278 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.998079 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:20.998080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:20.998080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:20.998088 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:20.999583 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:20.999950 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:20.999951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:20.999951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:21.000013 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:21.000214 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:21.000215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:21.000216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:21.000344 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:21.000570 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:21.000606 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 21:00:21.000681 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 21:00:21.001520 22663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d: No bootstrap required, opened a new log
I20260926 21:00:21.002115 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:00:21.003183 22637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "62c8fce264a449cca64793667863915d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46748
I20260926 21:00:21.003453 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.990804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38843 (local address 127.0.0.1:46538)
0926 21:00:20.990848 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:20.990859 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 21:00:20.990895 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:00:20.991156 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:00:20.991157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:00:20.991307 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:00:20.991346 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:20.991347 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.992862 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.992863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:20.993695 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:20.993698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.994236 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.994237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:20.994594 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.994971 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:00:20.994976 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:00:20.996746 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:00:21.002185 (+ 5439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:21.002186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:21.002186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:21.002394 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:21.002628 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:21.002629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:21.002629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:21.002702 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:21.003080 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:00:21.003081 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:00:21.003131 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 21:00:21.003209 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 21:00:21.003315 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 21:00:21.003660 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:20.990860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.0.98.60:38843)
0926 21:00:20.991022 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:20.991023 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:20.991030 (+ 7us) server_negotiation.cc:407] Connection header received
0926 21:00:20.991048 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:20.991048 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:20.991059 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:20.991125 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:20.991430 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.992796 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.993803 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:20.994139 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:20.994209 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:20.996865 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:20.996866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:20.996867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:20.996873 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:21.002106 (+ 5233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:21.002476 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:21.002477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:21.002478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:21.002548 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:21.002781 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:21.002782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:21.002782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:21.002978 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:21.003083 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:21.003256 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 21:00:21.003387 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":144,"thread_start_us":116,"threads_started":1}
I20260926 21:00:21.004129 22663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8fce264a449cca64793667863915d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } peers { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } } peers { permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } }
I20260926 21:00:21.004257 22663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c8fce264a449cca64793667863915d, State: Initialized, Role: FOLLOWER
I20260926 21:00:21.004354 22663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8fce264a449cca64793667863915d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } peers { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } } peers { permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } }
I20260926 21:00:21.004782 22663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:21.004746 22820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8fce264a449cca64793667863915d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } peers { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } } peers { permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } } }
I20260926 21:00:21.004791 22637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 21:00:21.004884 22820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:00:21.004976 22637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:21.005240 22813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1: Bootstrap starting.
I20260926 21:00:21.005841 22637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8fce264a449cca64793667863915d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } peers { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } } peers { permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } }
I20260926 21:00:21.005956 22813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:21.006115 22637 leader_election.cc:288] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [CANDIDATE]: Term 1 election: Requested vote from peers 41a74c468a4b4dca8938386e9c2aa72b (127.0.98.61:40265), e5235cfd3b3a4fd49e3d0ce1d716deb1 (127.0.98.60:38843)
W20260926 21:00:21.007062 22831 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:00:21.007140 22831 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:00:21.008564 22813 log.cc:824] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:21.008107 22595 master_runner.cc:186] Error getting master registration for 127.0.98.61:40265: OK, instance_id { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" instance_seqno: 1790456420813688 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 21:00:21.011168 22595 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 21:00:21.013159 22813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1: No bootstrap required, opened a new log
I20260926 21:00:21.015625 22813 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8fce264a449cca64793667863915d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } peers { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } } peers { permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } }
I20260926 21:00:21.015813 22813 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5235cfd3b3a4fd49e3d0ce1d716deb1, State: Initialized, Role: FOLLOWER
I20260926 21:00:21.015914 22813 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8fce264a449cca64793667863915d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } peers { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } } peers { permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } }
I20260926 21:00:21.016292 22813 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:21.016368 22787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62c8fce264a449cca64793667863915d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1"
I20260926 21:00:21.016571 22787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:21.017458 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:21.007622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47388 (local address 127.0.98.61:40265)
0926 21:00:21.008350 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:21.008353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:21.008379 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:00:21.008421 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:21.008421 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:21.008431 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:21.008495 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:21.009509 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:21.010216 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:21.011197 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:21.011818 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:21.011922 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:21.013629 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:21.013630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:21.013630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:21.013638 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:21.015635 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:21.016254 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:21.016255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:21.016255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:21.016319 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:21.016759 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:21.016760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:21.016761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:21.016931 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:21.017019 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:21.017264 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 21:00:21.017358 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701}
I20260926 21:00:21.017580 22787 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c8fce264a449cca64793667863915d in term 1.
I20260926 21:00:21.017910 22613 leader_election.cc:302] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c8fce264a449cca64793667863915d, e5235cfd3b3a4fd49e3d0ce1d716deb1; no voters:
I20260926 21:00:21.018111 22820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:00:21.018646 22820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [term 1 LEADER]: Becoming Leader. State: Replica: 62c8fce264a449cca64793667863915d, State: Running, Role: LEADER
I20260926 21:00:21.018779 22820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8fce264a449cca64793667863915d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } peers { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } } peers { permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } }
I20260926 21:00:21.019302 22820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62c8fce264a449cca64793667863915d. Latest consensus state: current_term: 1 leader_uuid: "62c8fce264a449cca64793667863915d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8fce264a449cca64793667863915d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } peers { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } } peers { permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } } }
I20260926 21:00:21.019410 22820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:21.019598 22855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:00:21.019819 22855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:00:21.020530 22841 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8fce264a449cca64793667863915d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } peers { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } } peers { permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } } }
I20260926 21:00:21.020669 22841 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 21:00:21.021703 22852 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:00:21.021771 22852 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:00:21.023633 22787 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1 [term 1 FOLLOWER]: Refusing update from remote peer 62c8fce264a449cca64793667863915d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:00:21.023947 22820 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:00:21.022892 22745 master_runner.cc:186] Error getting master registration for 127.0.98.61:40265: OK, instance_id { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" instance_seqno: 1790456420813688 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 21:00:21.026196 22745 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 21:00:21.026667 22841 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62c8fce264a449cca64793667863915d. Latest consensus state: current_term: 1 leader_uuid: "62c8fce264a449cca64793667863915d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8fce264a449cca64793667863915d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } peers { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } } peers { permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } } }
I20260926 21:00:21.026818 22841 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:00:21.027017 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:21.008135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46540 (local address 127.0.98.60:38843)
0926 21:00:21.008174 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:21.008176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:21.012073 (+ 3897us) server_negotiation.cc:407] Connection header received
0926 21:00:21.012520 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:21.012522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:21.012536 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:21.012569 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:21.018625 (+ 6056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:21.019329 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:21.021420 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:21.021714 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:21.021784 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:21.023533 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:21.023535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:21.023535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:21.023542 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:21.025517 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:21.025932 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:21.025934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:21.025934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:21.025997 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:21.026327 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:21.026328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:21.026329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:21.026496 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:21.026602 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:21.026825 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 21:00:21.026931 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 21:00:21.027978 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:21.007429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.0.98.62:46675)
0926 21:00:21.007477 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:21.007479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:21.007628 (+ 149us) server_negotiation.cc:407] Connection header received
0926 21:00:21.007981 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:21.007982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:21.007993 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:21.008025 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:21.010105 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:21.010895 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:21.013372 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:21.013699 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:21.013763 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:21.020259 (+ 6496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:21.020260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:21.020261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:21.020272 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:21.026132 (+ 5860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:21.026811 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:21.026812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:21.026812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:21.026892 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:21.027194 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:21.027195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:21.027195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:21.027348 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:21.027435 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:21.027777 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 21:00:21.027872 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 21:00:21.030162 22853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62c8fce264a449cca64793667863915d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8fce264a449cca64793667863915d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } peers { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } } peers { permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } } }
I20260926 21:00:21.030284 22853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:21.030928 22857 mvcc.cc:205] Tried to move back new op lower bound from 7333709500510015488 to 7333709500493963264. Current Snapshot: MvccSnapshot[applied={T|T < 7333709500510015488 or (T in {7333709500510015488})}]
I20260926 21:00:21.031008 22855 catalog_manager.cc:1347] Generated new cluster ID: 941a9d8e3dfe4134b2bbccd7cf9c2c81
I20260926 21:00:21.031059 22855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:00:21.031119 22841 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62c8fce264a449cca64793667863915d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8fce264a449cca64793667863915d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } peers { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } } peers { permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } } }
I20260926 21:00:21.031201 22841 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:00:21.047936 22855 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:00:21.048864 22855 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:00:21.054062 22738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } attempt: 3
Sep 26 21:00:21 dist-test-slave-fl7g krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 21:00:21.064188 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:21.054374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38843 (local address 127.0.0.1:46550)
0926 21:00:21.054474 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:21.054486 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 21:00:21.054527 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:00:21.054902 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:00:21.054903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:00:21.055101 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:00:21.055153 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:21.055155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:21.056138 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:21.056139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:21.057028 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:21.057030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:21.057608 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:21.057609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:21.057663 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:21.058177 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:00:21.058183 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:00:21.060799 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:00:21.062889 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:21.062890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:21.062890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:21.063130 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:21.063395 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:21.063395 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:21.063396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:21.063473 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:21.063833 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:00:21.063834 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:00:21.063899 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 21:00:21.063963 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 21:00:21.064076 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 21:00:21.064834 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:21.054613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.0.98.60:38843)
0926 21:00:21.054649 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:21.054651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:21.054658 (+ 7us) server_negotiation.cc:407] Connection header received
0926 21:00:21.054681 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:21.054681 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:21.054691 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:21.054736 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:21.055241 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:21.056037 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:21.057122 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:21.057465 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:21.057524 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:21.060898 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:21.060900 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:21.060900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:21.060908 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:21.062767 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:21.063225 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:21.063226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:21.063227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:21.063303 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:21.063562 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:21.063563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:21.063563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:21.063717 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:21.063787 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:21.064651 (+ 864us) server_negotiation.cc:299] Negotiation successful
0926 21:00:21.064741 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 21:00:21.066954 22738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b: Bootstrap starting.
I20260926 21:00:21.067734 22738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:21.068022 22738 log.cc:824] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:21.069347 22855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d: Generated new TSK 0
I20260926 21:00:21.069872 22855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:00:21.070503 22738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b: No bootstrap required, opened a new log
I20260926 21:00:21.072573 22738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8fce264a449cca64793667863915d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } peers { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } } peers { permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } }
I20260926 21:00:21.072726 22738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a74c468a4b4dca8938386e9c2aa72b, State: Initialized, Role: FOLLOWER
I20260926 21:00:21.072803 22738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8fce264a449cca64793667863915d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } peers { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } } peers { permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } }
I20260926 21:00:21.073186 22738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:21.073249 22712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62c8fce264a449cca64793667863915d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "41a74c468a4b4dca8938386e9c2aa72b"
I20260926 21:00:21.073390 22712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:21.074419 22712 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c8fce264a449cca64793667863915d in term 1.
I20260926 21:00:21.074914 22711 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b [term 1 FOLLOWER]: Refusing update from remote peer 62c8fce264a449cca64793667863915d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:00:21.075410 22820 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:00:21.079860 22863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62c8fce264a449cca64793667863915d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8fce264a449cca64793667863915d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } peers { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } } peers { permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } } }
I20260926 21:00:21.079969 22863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:00:21.080020 22863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62c8fce264a449cca64793667863915d. Latest consensus state: current_term: 1 leader_uuid: "62c8fce264a449cca64793667863915d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8fce264a449cca64793667863915d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } peers { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } } peers { permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } } }
I20260926 21:00:21.080080 22863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b [sys.catalog]: This master's current role is: FOLLOWER
W20260926 21:00:21.080106 22874 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:00:21.080156 22874 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:00:21.080112 22863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62c8fce264a449cca64793667863915d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8fce264a449cca64793667863915d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } peers { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } } peers { permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } } }
I20260926 21:00:21.080266 22863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:00:21.085111 22876 mvcc.cc:205] Tried to move back new op lower bound from 7333709500680024064 to 7333709500493963264. Current Snapshot: MvccSnapshot[applied={T|T < 7333709500510015488}]
W20260926 21:00:21.095577 22743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:21.093905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.0.98.61:40265)
0926 21:00:21.093959 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:21.093962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:21.094868 (+ 906us) server_negotiation.cc:407] Connection header received
0926 21:00:21.094978 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:21.094979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:21.094991 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:21.095024 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:21.095490 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47402: BlockingRecv error: recv got EOF from 127.0.0.1:47402 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260926 21:00:21.096619 22818 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:21.094254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46554 (local address 127.0.98.60:38843)
0926 21:00:21.094296 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:21.094298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:21.096024 (+ 1726us) server_negotiation.cc:407] Connection header received
0926 21:00:21.096122 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:21.096123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:21.096134 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:21.096163 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:21.096554 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46554: BlockingRecv error: recv got EOF from 127.0.0.1:46554 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 21:00:21.097546 22666 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:21.094603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46764 (local address 127.0.98.62:46675)
0926 21:00:21.094647 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:21.094649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:21.096955 (+ 2306us) server_negotiation.cc:407] Connection header received
0926 21:00:21.097056 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:21.097058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:21.097069 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:21.097099 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:21.097478 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46764: BlockingRecv error: recv got EOF from 127.0.0.1:46764 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 21:00:21.105443 22743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:21.100779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47404 (local address 127.0.98.61:40265)
0926 21:00:21.100819 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:21.100821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:21.104762 (+ 3941us) server_negotiation.cc:407] Connection header received
0926 21:00:21.104900 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:21.104901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:21.104927 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:21.104960 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:21.105383 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47404: BlockingRecv error: recv got EOF from 127.0.0.1:47404 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 21:00:21.108845 22743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:21.107944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47406 (local address 127.0.98.61:40265)
0926 21:00:21.107984 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:21.107986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:21.108249 (+ 263us) server_negotiation.cc:407] Connection header received
0926 21:00:21.108348 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:21.108349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:21.108373 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:21.108426 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:21.108786 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47406: BlockingRecv error: recv got EOF from 127.0.0.1:47406 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 21:00:21.113075 22743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:21.112122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.0.98.61:40265)
0926 21:00:21.112161 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:21.112163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:21.112513 (+ 350us) server_negotiation.cc:407] Connection header received
0926 21:00:21.112618 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:21.112619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:21.112630 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:21.112661 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:21.113019 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47412: BlockingRecv error: recv got EOF from 127.0.0.1:47412 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 21:00:21.119230 22743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:21.118392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.0.98.61:40265)
0926 21:00:21.118431 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:21.118433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:21.118684 (+ 251us) server_negotiation.cc:407] Connection header received
0926 21:00:21.118781 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:21.118782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:21.118793 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:21.118821 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:21.119175 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47414: BlockingRecv error: recv got EOF from 127.0.0.1:47414 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 21:00:21.129616 22743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:21.128663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47428 (local address 127.0.98.61:40265)
0926 21:00:21.128709 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:21.128711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:21.128995 (+ 284us) server_negotiation.cc:407] Connection header received
0926 21:00:21.129102 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:21.129104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:21.129115 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:21.129148 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:21.129556 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47428: BlockingRecv error: recv got EOF from 127.0.0.1:47428 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 21:00:21.148128 22743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:21.147133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47432 (local address 127.0.98.61:40265)
0926 21:00:21.147180 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:21.147183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:21.147498 (+ 315us) server_negotiation.cc:407] Connection header received
0926 21:00:21.147608 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:21.147609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:21.147621 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:21.147654 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:21.148064 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47432: BlockingRecv error: recv got EOF from 127.0.0.1:47432 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 21:00:21.182521 22743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:21.181616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47438 (local address 127.0.98.61:40265)
0926 21:00:21.181661 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:21.181664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:21.181936 (+ 272us) server_negotiation.cc:407] Connection header received
0926 21:00:21.182036 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:21.182037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:21.182048 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:21.182078 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:21.182459 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47438: BlockingRecv error: recv got EOF from 127.0.0.1:47438 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 21:00:21.248894 22743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:21.248039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47450 (local address 127.0.98.61:40265)
0926 21:00:21.248111 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:21.248113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:21.248243 (+ 130us) server_negotiation.cc:407] Connection header received
0926 21:00:21.248331 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:21.248332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:21.248347 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:21.248409 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:21.248822 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47450: BlockingRecv error: recv got EOF from 127.0.0.1:47450 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 21:00:21.379113 22743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:21.378121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47462 (local address 127.0.98.61:40265)
0926 21:00:21.378230 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:21.378233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:21.378352 (+ 119us) server_negotiation.cc:407] Connection header received
0926 21:00:21.378487 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:21.378488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:21.378513 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:21.378606 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:21.379028 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47462: BlockingRecv error: recv got EOF from 127.0.0.1:47462 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:00:21.526362 22820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62c8fce264a449cca64793667863915d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8fce264a449cca64793667863915d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46675 } } peers { permanent_uuid: "41a74c468a4b4dca8938386e9c2aa72b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40265 } } peers { permanent_uuid: "e5235cfd3b3a4fd49e3d0ce1d716deb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38843 } } }
I20260926 21:00:21.526544 22820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62c8fce264a449cca64793667863915d [sys.catalog]: This master's current role is: LEADER
W20260926 21:00:21.637529 22743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:21.636561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47466 (local address 127.0.98.61:40265)
0926 21:00:21.636642 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:21.636643 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:21.636654 (+ 11us) server_negotiation.cc:407] Connection header received
0926 21:00:21.636821 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:21.636822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:21.636835 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:21.636897 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:21.637435 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47466: BlockingRecv error: recv got EOF from 127.0.0.1:47466 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 21:00:22.022228 22852 catalog_manager.cc:1259] Loaded cluster ID: 941a9d8e3dfe4134b2bbccd7cf9c2c81
I20260926 21:00:22.022310 22852 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1: loading cluster ID for follower catalog manager: success
I20260926 21:00:22.023182 22852 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1: acquiring CA information for follower catalog manager: success
I20260926 21:00:22.023546 22852 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e5235cfd3b3a4fd49e3d0ce1d716deb1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 21:00:22.080742 22874 catalog_manager.cc:1259] Loaded cluster ID: 941a9d8e3dfe4134b2bbccd7cf9c2c81
I20260926 21:00:22.080825 22874 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b: loading cluster ID for follower catalog manager: success
I20260926 21:00:22.081699 22874 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b: acquiring CA information for follower catalog manager: success
I20260926 21:00:22.082032 22874 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 41a74c468a4b4dca8938386e9c2aa72b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 21:00:22.153579 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:22.150639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47470 (local address 127.0.98.61:40265)
0926 21:00:22.150771 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:22.150773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:22.150886 (+ 113us) server_negotiation.cc:407] Connection header received
0926 21:00:22.150927 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:22.150928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:22.150984 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:22.151077 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:00:22.151224 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:22.151808 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:22.152720 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:22.152952 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:22.153024 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:22.153172 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:00:22.153290 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:00:22.153359 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:22.153399 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 21:00:22.153421 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260926 21:00:22.157975 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:22.155656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.0.98.60:38843)
0926 21:00:22.155776 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:22.155778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:22.155812 (+ 34us) server_negotiation.cc:407] Connection header received
0926 21:00:22.155880 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:22.155881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:22.155916 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:22.155985 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:00:22.156088 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:22.156635 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:22.157328 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:22.157508 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:22.157556 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:22.157681 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:00:22.157803 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:00:22.157861 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:22.157895 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 21:00:22.157911 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68,"thread_start_us":46,"threads_started":1}
I20260926 21:00:22.162703 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:22.159609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47478 (local address 127.0.98.61:40265)
0926 21:00:22.159651 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:22.159653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:22.159856 (+ 203us) server_negotiation.cc:407] Connection header received
0926 21:00:22.159948 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:22.159949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:22.159960 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:22.160018 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:00:22.160123 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:22.160827 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:22.161920 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:22.162173 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:22.162223 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:22.162411 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:00:22.162444 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:00:22.162517 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:22.162581 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 21:00:22.162605 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 21:00:22.163008 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:22.159901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46576 (local address 127.0.98.60:38843)
0926 21:00:22.159936 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:22.159938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:22.160072 (+ 134us) server_negotiation.cc:407] Connection header received
0926 21:00:22.160095 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:22.160097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:22.160108 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:22.160150 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:00:22.160299 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:22.160935 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:22.162254 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:22.162496 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:22.162554 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:22.162718 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:00:22.162747 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:00:22.162791 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:22.162897 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 21:00:22.162920 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 21:00:22.163259 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:22.159614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46774 (local address 127.0.98.62:46675)
0926 21:00:22.159772 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:22.159775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:22.159948 (+ 173us) server_negotiation.cc:407] Connection header received
0926 21:00:22.159992 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:22.159993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:22.160044 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:22.160131 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:00:22.160626 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:22.161267 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:22.162053 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:22.162300 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:22.162373 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:22.162776 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:00:22.162925 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:00:22.162989 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:22.163145 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 21:00:22.163175 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":46,"threads_started":1}
I20260926 21:00:22.164888 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 22595
I20260926 21:00:22.170647 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 22670
I20260926 21:00:22.176651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 22745
2026-09-26T21:00:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2821 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:00:22 dist-test-slave-fl7g krb5kdc[22965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:00:22 dist-test-slave-fl7g krb5kdc[22965](info): set up 2 sockets
Sep 26 21:00:22 dist-test-slave-fl7g krb5kdc[22965](info): commencing operation
krb5kdc: starting...
W20260926 21:00:23.208777 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/test-user.keytab.
Sep 26 21:00:23 dist-test-slave-fl7g krb5kdc[22965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:00:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:00:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 21:00:23.335949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37833
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37833,127.0.98.61:33831,127.0.98.60:38151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:00:23.417814 22978 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 21:00:23.418112 22978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:23.418144 22978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:23.418191 22978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:23.419442 22978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:23.419523 22978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:23.419548 22978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:23.419570 22978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:23.419598 22978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:23.421228 22978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37833,127.0.98.61:33831,127.0.98.60:38151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:23.421538 22978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:23.421785 22978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:23.427008 22986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:23.426997 22984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:23.427001 22983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:23.427330 22978 server_base.cc:1034] running on GCE node
I20260926 21:00:23.427695 22978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:23.427989 22978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:23.429176 22978 hybrid_clock.cc:648] HybridClock initialized: now 1790456423429154 us; error 45 us; skew 500 ppm
Sep 26 21:00:23 dist-test-slave-fl7g krb5kdc[22965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:23.438079 22978 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 21:00:23.438697 22978 webserver.cc:460] Webserver started at http://127.0.98.62:35455/ using document root <none> and password file <none>
I20260926 21:00:23.438987 22978 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:23.439040 22978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:23.439194 22978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:23.440382 22978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-0/data/instance:
uuid: "412c9a44e7974194a3a9b2fb475da533"
format_stamp: "Formatted at 2026-09-26 21:00:23 on dist-test-slave-fl7g"
server_key: "e896a6a72865f12657dd209854d733fc"
server_key_iv: "0da93463e141409d0f90235e68feedeb"
server_key_version: "encryptionkey@0"
I20260926 21:00:23.440881 22978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-0/wal/instance:
uuid: "412c9a44e7974194a3a9b2fb475da533"
format_stamp: "Formatted at 2026-09-26 21:00:23 on dist-test-slave-fl7g"
server_key: "e896a6a72865f12657dd209854d733fc"
server_key_iv: "0da93463e141409d0f90235e68feedeb"
server_key_version: "encryptionkey@0"
I20260926 21:00:23.442384 22978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 21:00:23.443295 22993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:23.443523 22978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 21:00:23.443619 22978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-0/wal
uuid: "412c9a44e7974194a3a9b2fb475da533"
format_stamp: "Formatted at 2026-09-26 21:00:23 on dist-test-slave-fl7g"
server_key: "e896a6a72865f12657dd209854d733fc"
server_key_iv: "0da93463e141409d0f90235e68feedeb"
server_key_version: "encryptionkey@0"
I20260926 21:00:23.443701 22978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:23.453877 22978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:23.456099 22978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:23.456254 22978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:23.461050 22978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37833
I20260926 21:00:23.461076 23045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37833 every 8 connection(s)
I20260926 21:00:23.461529 22978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-0/data/info.pb
I20260926 21:00:23.462592 23046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:23.464154 23046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:00:23 dist-test-slave-fl7g krb5kdc[22965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 21:00:23.471283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 22978
I20260926 21:00:23.471421 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-0/wal/instance
I20260926 21:00:23.471668 392 external_mini_cluster.cc:1428] Setting key c2bc8c8d024fdb0c7df70ab27efd19d6
I20260926 21:00:23.474561 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.464952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37833 (local address 127.0.0.1:37104)
0926 21:00:23.465122 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.465165 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 21:00:23.465630 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:00:23.465904 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:00:23.465906 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:00:23.466169 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:00:23.466333 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:23.466337 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.467344 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.467345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:23.468053 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:23.468055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.468555 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.468556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:23.468610 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.469158 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:00:23.469167 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:00:23.470574 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:00:23.472814 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:23.472821 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:23.472823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:23.473043 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:23.473375 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:23.473377 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:23.473377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:23.473502 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:23.473858 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:00:23.473864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:00:23.473945 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 21:00:23.474065 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 21:00:23.474377 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":117,"thread_start_us":87,"threads_started":1}
I20260926 21:00:23.474568 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.464980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37104 (local address 127.0.98.62:37833)
0926 21:00:23.465162 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.465172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.465264 (+ 92us) server_negotiation.cc:407] Connection header received
0926 21:00:23.465766 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.465768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.465785 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.465882 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.466416 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.467259 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.468133 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.468468 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.468557 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.470682 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:23.470690 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:23.470691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:23.470697 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:23.472713 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.473164 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.473168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.473169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.473292 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.473569 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.473570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.473570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.473791 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:23.473855 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:23.474088 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 21:00:23.474278 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":144,"thread_start_us":116,"threads_started":1}
I20260926 21:00:23.475733 23046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:00:23.476115 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.476002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33831 (local address 127.0.0.1:60382)
0926 21:00:23.476051 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.476093 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 21:00:23.477465 22994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
I20260926 21:00:23.477967 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.477852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33831 (local address 127.0.0.1:60398)
0926 21:00:23.477913 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.477944 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 21:00:23.478102 23046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33831: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 21:00:23.499651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33831
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37833,127.0.98.61:33831,127.0.98.60:38151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:00:23.527354 23046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } attempt: 1
I20260926 21:00:23.527987 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.527798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33831 (local address 127.0.0.1:60412)
0926 21:00:23.527893 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.527952 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 21:00:23.528468 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.528322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33831 (local address 127.0.0.1:60424)
0926 21:00:23.528370 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.528397 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 21:00:23.528607 23046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33831: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
W20260926 21:00:23.585894 23053 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 21:00:23.586130 23053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:23.586149 23053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:23.586172 23053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:23.587718 23053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:23.587781 23053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:23.587796 23053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:23.587809 23053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:23.587821 23053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:23.589519 23053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37833,127.0.98.61:33831,127.0.98.60:38151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:23.589798 23053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:23.590065 23053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:23.595067 23059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:23.595113 23058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:23.595067 23061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:23.595302 23053 server_base.cc:1034] running on GCE node
I20260926 21:00:23.595533 23053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:23.595824 23053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:23.597005 23053 hybrid_clock.cc:648] HybridClock initialized: now 1790456423596981 us; error 42 us; skew 500 ppm
I20260926 21:00:23.596773 23046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } attempt: 2
I20260926 21:00:23.597436 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.597281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33831 (local address 127.0.0.1:60426)
0926 21:00:23.597359 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.597396 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 21:00:23.597836 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.597726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33831 (local address 127.0.0.1:60436)
0926 21:00:23.597783 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.597816 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 21:00:23.597977 23046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33831: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
Sep 26 21:00:23 dist-test-slave-fl7g krb5kdc[22965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:23.599680 23053 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 21:00:23.600189 23053 webserver.cc:460] Webserver started at http://127.0.98.61:34541/ using document root <none> and password file <none>
I20260926 21:00:23.600392 23053 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:23.600502 23053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:23.600598 23053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:23.601650 23053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-1/data/instance:
uuid: "004b8632c12344ff8356b09f61498b62"
format_stamp: "Formatted at 2026-09-26 21:00:23 on dist-test-slave-fl7g"
server_key: "98f476580cf557779fe9758029a19d48"
server_key_iv: "165052601e045d0933e2c4e8c58d135b"
server_key_version: "encryptionkey@0"
I20260926 21:00:23.601964 23053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-1/wal/instance:
uuid: "004b8632c12344ff8356b09f61498b62"
format_stamp: "Formatted at 2026-09-26 21:00:23 on dist-test-slave-fl7g"
server_key: "98f476580cf557779fe9758029a19d48"
server_key_iv: "165052601e045d0933e2c4e8c58d135b"
server_key_version: "encryptionkey@0"
I20260926 21:00:23.603250 23053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 21:00:23.604004 23068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:23.604223 23053 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 21:00:23.604329 23053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-1/wal
uuid: "004b8632c12344ff8356b09f61498b62"
format_stamp: "Formatted at 2026-09-26 21:00:23 on dist-test-slave-fl7g"
server_key: "98f476580cf557779fe9758029a19d48"
server_key_iv: "165052601e045d0933e2c4e8c58d135b"
server_key_version: "encryptionkey@0"
I20260926 21:00:23.604409 23053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:23.621524 23053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:23.624099 23053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:23.624274 23053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:23.628401 23053 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33831
I20260926 21:00:23.628486 23120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33831 every 8 connection(s)
I20260926 21:00:23.629062 23053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-1/data/info.pb
I20260926 21:00:23.630084 23121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:23.631652 23121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:00:23.635673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 23053
I20260926 21:00:23.635834 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-1/wal/instance
I20260926 21:00:23.636057 392 external_mini_cluster.cc:1428] Setting key b2de5c7226df7d5db5c35faa038bb762
Sep 26 21:00:23 dist-test-slave-fl7g krb5kdc[22965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 21:00:23.641446 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.632446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37116 (local address 127.0.98.62:37833)
0926 21:00:23.632525 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.632527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.632670 (+ 143us) server_negotiation.cc:407] Connection header received
0926 21:00:23.633126 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.633126 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.633138 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.633202 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.633769 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.634265 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.635278 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.635516 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.635603 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.638216 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:23.638218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:23.638218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:23.638227 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:23.639754 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.640311 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.640312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.640312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.640371 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.640727 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.640728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.640728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.640851 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:23.640905 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:23.641301 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 21:00:23.641378 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 21:00:23.641780 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.632349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37833 (local address 127.0.0.1:37116)
0926 21:00:23.632539 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.632583 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 21:00:23.633045 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:00:23.633254 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:00:23.633258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:00:23.633507 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:00:23.633697 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:23.633701 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.634348 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.634349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:23.635194 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:23.635196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.635640 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.635641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:23.635773 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.636507 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:00:23.636515 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:00:23.638095 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:00:23.639861 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:23.639871 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:23.639875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:23.640219 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:23.640489 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:23.640490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:23.640490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:23.640655 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:23.640923 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:00:23.640931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:00:23.641132 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 21:00:23.641275 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 21:00:23.641480 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":110,"threads_started":1}
I20260926 21:00:23.642488 23121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 21:00:23 dist-test-slave-fl7g krb5kdc[22965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 21:00:23.650693 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.642756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33831 (local address 127.0.0.1:60450)
0926 21:00:23.642993 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.643004 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 21:00:23.643059 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:00:23.643199 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:00:23.643200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:00:23.643363 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:00:23.643423 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:23.643425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.644306 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.644307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:23.644932 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:23.644935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.645392 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.645393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:23.645451 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.645806 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:00:23.645810 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:00:23.647637 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:00:23.649660 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:23.649661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:23.649661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:23.649840 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:23.650057 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:23.650057 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:23.650058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:23.650110 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:23.650438 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:00:23.650438 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:00:23.650475 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 21:00:23.650531 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 21:00:23.650617 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260926 21:00:23.650858 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.642832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.0.98.61:33831)
0926 21:00:23.643008 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.643012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.643052 (+ 40us) server_negotiation.cc:407] Connection header received
0926 21:00:23.643116 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.643118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.643129 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.643184 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.643493 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.644177 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.645001 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.645327 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.645383 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.647739 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:23.647746 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:23.647747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:23.647755 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:23.649559 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.649922 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.649925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.649925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.649990 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.650168 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.650169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.650169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.650370 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:23.650456 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:23.650566 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 21:00:23.650693 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":150,"thread_start_us":106,"threads_started":1}
I20260926 21:00:23.651502 23121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:00:23.651901 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.651757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38151 (local address 127.0.0.1:41028)
0926 21:00:23.651810 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.651867 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 21:00:23.652055 23072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38151: connect: Connection refused (error 111)
I20260926 21:00:23.652577 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.652482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38151 (local address 127.0.0.1:41034)
0926 21:00:23.652533 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.652559 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 21:00:23.652715 23121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38151: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38151: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 21:00:23.664016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVxazDW/build/release/bin/kudu
/tmp/dist-test-taskVxazDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38151
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37833,127.0.98.61:33831,127.0.98.60:38151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:00:23.689146 23046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } attempt: 3
Sep 26 21:00:23 dist-test-slave-fl7g krb5kdc[22965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 21:00:23.696663 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.689547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33831 (local address 127.0.0.1:60458)
0926 21:00:23.689602 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.689618 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:00:23.689692 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:00:23.689859 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:00:23.689860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:00:23.690030 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:00:23.690094 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:23.690095 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.690785 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.690787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:23.691344 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:23.691346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.691739 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.691740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:23.691827 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.692372 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:00:23.692377 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:00:23.693915 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:00:23.695551 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:23.695552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:23.695553 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:23.695734 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:23.695944 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:23.695945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:23.695945 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:23.696012 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:23.696271 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:00:23.696272 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:00:23.696325 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 21:00:23.696449 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 21:00:23.696561 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 21:00:23.696663 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.689625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.0.98.61:33831)
0926 21:00:23.689695 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.689697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.689708 (+ 11us) server_negotiation.cc:407] Connection header received
0926 21:00:23.689760 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.689761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.689776 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.689839 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.690182 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.690695 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.691430 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.691661 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.691741 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.694018 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:23.694020 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:23.694020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:23.694031 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:23.695460 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.695815 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.695816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.695817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.695875 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.696075 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.696076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.696077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.696195 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:23.696276 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:23.696476 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 21:00:23.696561 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 21:00:23.697157 23046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:00:23.697543 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.697430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38151 (local address 127.0.0.1:41046)
0926 21:00:23.697482 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.697514 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 21:00:23.697855 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.697762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38151 (local address 127.0.0.1:41062)
0926 21:00:23.697811 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.697834 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 21:00:23.697954 23046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38151: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38151: connect: Connection refused (error 111)
I20260926 21:00:23.701996 23121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } attempt: 1
I20260926 21:00:23.702503 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.702372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38151 (local address 127.0.0.1:41078)
0926 21:00:23.702430 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.702463 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 21:00:23.702922 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.702837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38151 (local address 127.0.0.1:41084)
0926 21:00:23.702882 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.702906 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 21:00:23.703032 23121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38151: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38151: connect: Connection refused (error 111)
I20260926 21:00:23.729123 23046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } attempt: 1
I20260926 21:00:23.729604 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.729465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38151 (local address 127.0.0.1:41088)
0926 21:00:23.729533 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.729575 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 21:00:23.729975 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.729877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38151 (local address 127.0.0.1:41104)
0926 21:00:23.729923 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.729951 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 21:00:23.730062 23046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38151: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38151: connect: Connection refused (error 111)
W20260926 21:00:23.748314 23128 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 21:00:23.748560 23128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:00:23.748579 23128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:00:23.748634 23128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:00:23.749822 23128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:00:23.749864 23128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:00:23.749879 23128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:00:23.749892 23128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:00:23.749943 23128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:00:23.751426 23128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37833,127.0.98.61:33831,127.0.98.60:38151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 21:00:23.751680 23128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:00:23.751924 23128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:00:23.756867 23136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:23.757000 23128 server_base.cc:1034] running on GCE node
W20260926 21:00:23.757071 23133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:00:23.756888 23134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:00:23.757392 23128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:00:23.757743 23128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:00:23.758921 23128 hybrid_clock.cc:648] HybridClock initialized: now 1790456423758910 us; error 32 us; skew 500 ppm
Sep 26 21:00:23 dist-test-slave-fl7g krb5kdc[22965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:00:23.761745 23128 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 21:00:23.762197 23128 webserver.cc:460] Webserver started at http://127.0.98.60:39135/ using document root <none> and password file <none>
I20260926 21:00:23.762439 23128 fs_manager.cc:359] Metadata directory not provided
I20260926 21:00:23.762508 23128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:00:23.762662 23128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:00:23.763743 23128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-2/data/instance:
uuid: "f6f7f9567a314a92a764aa0744cf7562"
format_stamp: "Formatted at 2026-09-26 21:00:23 on dist-test-slave-fl7g"
server_key: "6923f24c9da44fef8aa6bc7fe7ca1e8f"
server_key_iv: "2b63f9dc5c65d139a8d1f9b8daebe306"
server_key_version: "encryptionkey@0"
I20260926 21:00:23.764130 23128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-2/wal/instance:
uuid: "f6f7f9567a314a92a764aa0744cf7562"
format_stamp: "Formatted at 2026-09-26 21:00:23 on dist-test-slave-fl7g"
server_key: "6923f24c9da44fef8aa6bc7fe7ca1e8f"
server_key_iv: "2b63f9dc5c65d139a8d1f9b8daebe306"
server_key_version: "encryptionkey@0"
I20260926 21:00:23.765565 23128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 21:00:23.766340 23143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:00:23.766512 23128 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 21:00:23.766611 23128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-2/wal
uuid: "f6f7f9567a314a92a764aa0744cf7562"
format_stamp: "Formatted at 2026-09-26 21:00:23 on dist-test-slave-fl7g"
server_key: "6923f24c9da44fef8aa6bc7fe7ca1e8f"
server_key_iv: "2b63f9dc5c65d139a8d1f9b8daebe306"
server_key_version: "encryptionkey@0"
I20260926 21:00:23.766695 23128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:00:23.771219 23121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } attempt: 2
I20260926 21:00:23.771661 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.771517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38151 (local address 127.0.0.1:41118)
0926 21:00:23.771579 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.771622 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 21:00:23.772104 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.771971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38151 (local address 127.0.0.1:41120)
0926 21:00:23.772041 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.772072 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 21:00:23.772251 23121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38151: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38151: connect: Connection refused (error 111)
I20260926 21:00:23.776139 23128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:00:23.778463 23128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:00:23.778653 23128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:00:23.782932 23128 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38151
I20260926 21:00:23.782965 23195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38151 every 8 connection(s)
I20260926 21:00:23.783332 23128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-2/data/info.pb
I20260926 21:00:23.784358 23196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:00:23.786019 23196 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:00:23.788897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVxazDW/build/release/bin/kudu as pid 23128
I20260926 21:00:23.789008 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVxazDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790456150796563-392-0/minicluster-data/master-2/wal/instance
I20260926 21:00:23.789192 392 external_mini_cluster.cc:1428] Setting key 4309d866b78e65c5a08c9655cde034a5
Sep 26 21:00:23 dist-test-slave-fl7g krb5kdc[22965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 21:00:23 dist-test-slave-fl7g krb5kdc[22965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 21:00:23.796655 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.786723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37122 (local address 127.0.98.62:37833)
0926 21:00:23.786798 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.786800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.786980 (+ 180us) server_negotiation.cc:407] Connection header received
0926 21:00:23.787605 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.787607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.787618 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.787686 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.788470 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.789188 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.790537 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.790855 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.790911 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.793514 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:23.793515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:23.793516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:23.793524 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:23.794981 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.795517 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.795518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.795518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.795573 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.795879 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.795880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.795881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.796022 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:23.796120 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:23.796446 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 21:00:23.796536 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 21:00:23.796844 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.786669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37833 (local address 127.0.0.1:37122)
0926 21:00:23.786846 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.786889 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 21:00:23.787499 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:00:23.787812 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:00:23.787816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:00:23.788102 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:00:23.788315 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:23.788320 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.789297 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.789298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:23.790433 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:23.790444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.790971 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.790972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:23.791109 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.791696 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:00:23.791706 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:00:23.793385 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:00:23.795085 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:23.795092 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:23.795095 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:23.795436 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:23.795645 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:23.795646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:23.795646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:23.795791 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:23.796113 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:00:23.796120 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:00:23.796264 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 21:00:23.796402 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 21:00:23.796601 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260926 21:00:23.797567 23196 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:00:23.798803 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.789649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.0.98.60:38151)
0926 21:00:23.789858 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.789863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.789884 (+ 21us) server_negotiation.cc:407] Connection header received
0926 21:00:23.789906 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.789909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.789928 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.790016 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.790737 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.791755 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.792584 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.792924 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.793013 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.795491 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:23.795498 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:23.795499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:23.795507 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:23.797424 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.797949 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.797953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.797953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.798021 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.798244 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.798245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.798245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.798431 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:23.798571 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:23.798663 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 21:00:23.798728 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":111,"threads_started":1}
Sep 26 21:00:23 dist-test-slave-fl7g krb5kdc[22965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 21:00:23 dist-test-slave-fl7g krb5kdc[22965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 21:00:23.804955 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.797856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33831 (local address 127.0.0.1:60466)
0926 21:00:23.797897 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.797910 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 21:00:23.797951 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:00:23.798101 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:00:23.798102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:00:23.798278 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:00:23.798324 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:23.798326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.799261 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.799262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:23.799943 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:23.799946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.800377 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.800378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:23.800474 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.800915 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:00:23.800919 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:00:23.802133 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:00:23.803829 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:23.803830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:23.803831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:23.804044 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:23.804238 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:23.804238 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:23.804239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:23.804305 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:23.804611 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:00:23.804612 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:00:23.804666 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 21:00:23.804747 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 21:00:23.804845 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 21:00:23.804987 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.797916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60466 (local address 127.0.98.61:33831)
0926 21:00:23.797960 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.797962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.797973 (+ 11us) server_negotiation.cc:407] Connection header received
0926 21:00:23.798017 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.798018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.798026 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.798094 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.798412 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.799177 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.800039 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.800281 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.800378 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.802224 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:23.802226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:23.802226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:23.802235 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:23.803746 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.804123 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.804124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.804124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.804174 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.804373 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.804374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.804374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.804537 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:23.804595 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:23.804790 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 21:00:23.804883 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 21:00:23.805234 23046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } attempt: 2
I20260926 21:00:23.805480 23196 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:00:23.808116 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.799771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.0.98.62:37833)
0926 21:00:23.799816 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.799818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.799826 (+ 8us) server_negotiation.cc:407] Connection header received
0926 21:00:23.799859 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.799859 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.799868 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.799914 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.800384 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.801054 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.801739 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.801972 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.802030 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.804876 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:23.804878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:23.804878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:23.804885 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:23.806865 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.807298 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.807299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.807299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.807356 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.807543 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.807544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.807544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.807676 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:23.807887 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:23.807926 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 21:00:23.808019 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 21:00:23 dist-test-slave-fl7g krb5kdc[22965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 21:00:23 dist-test-slave-fl7g krb5kdc[22965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 21:00:23 dist-test-slave-fl7g krb5kdc[22965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 21:00:23.814546 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.805722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38151 (local address 127.0.0.1:41134)
0926 21:00:23.805903 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.805912 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 21:00:23.805973 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:00:23.806220 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:00:23.806221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:00:23.806379 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:00:23.806417 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:23.806419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.807434 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.807436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:23.808321 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:23.808323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.808932 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.808933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:23.808987 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.809363 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:00:23.809368 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:00:23.811012 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:00:23.813277 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:23.813278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:23.813279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:23.813543 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:23.813766 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:23.813767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:23.813767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:23.813833 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:23.814210 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:00:23.814210 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:00:23.814253 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 21:00:23.814315 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 21:00:23.814425 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260926 21:00:23.814862 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.805463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38151 (local address 127.0.0.1:41130)
0926 21:00:23.805596 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.805607 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 21:00:23.805644 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:00:23.805875 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:00:23.805876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:00:23.806085 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:00:23.806128 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:23.806130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.807024 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.807025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:23.807768 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:23.807769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.808471 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.808472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:23.808530 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.809026 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:00:23.809031 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:00:23.811016 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:00:23.813795 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:23.813797 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:23.813797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:23.814036 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:23.814245 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:23.814246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:23.814246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:23.814315 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:23.814571 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:00:23.814572 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:00:23.814621 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 21:00:23.814670 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 21:00:23.814765 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260926 21:00:23.814886 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.805784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41134 (local address 127.0.98.60:38151)
0926 21:00:23.806073 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.806075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.806081 (+ 6us) server_negotiation.cc:407] Connection header received
0926 21:00:23.806097 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.806097 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.806105 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.806217 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.806504 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.807254 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.808396 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.808709 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.808895 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.811105 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:23.811106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:23.811106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:23.811113 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:23.813180 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.813626 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.813627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.813628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.813688 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.813906 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.813907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.813907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.814126 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:23.814188 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:23.814677 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 21:00:23.814757 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":100,"threads_started":1}
I20260926 21:00:23.815721 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.805594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.0.98.60:38151)
0926 21:00:23.805764 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.805766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.805773 (+ 7us) server_negotiation.cc:407] Connection header received
0926 21:00:23.805790 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.805791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.805801 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.805869 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.806213 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.806933 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.807899 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.808344 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.808472 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.811104 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:23.811105 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:23.811105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:23.811113 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:23.813693 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.814116 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.814117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.814118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.814175 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.814385 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.814386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.814387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.814494 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:23.815531 (+ 1037us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:23.815572 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 21:00:23.815646 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260926 21:00:23.816778 23196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562: Bootstrap starting.
I20260926 21:00:23.817399 23196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:23.817551 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.808779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60470 (local address 127.0.98.61:33831)
0926 21:00:23.808821 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.808823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.808831 (+ 8us) server_negotiation.cc:407] Connection header received
0926 21:00:23.808901 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.808902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.808913 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.808978 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.809615 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.810373 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.811084 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.811313 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.811403 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.814384 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:23.814385 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:23.814386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:23.814393 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:23.816092 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.816592 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.816594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.816594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.816652 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.816861 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.816862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.816862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.817015 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:23.817158 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:23.817379 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 21:00:23.817463 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 21:00:23.817755 23196 log.cc:824] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:23.817905 23046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533: Bootstrap starting.
I20260926 21:00:23.818488 23196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562: No bootstrap required, opened a new log
I20260926 21:00:23.818548 23046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:23.818842 23046 log.cc:824] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:23.819654 23046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533: No bootstrap required, opened a new log
I20260926 21:00:23.820588 23196 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } peers { permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } }
I20260926 21:00:23.820789 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:00:23.820778 23196 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f7f9567a314a92a764aa0744cf7562, State: Initialized, Role: FOLLOWER
I20260926 21:00:23.820868 23196 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } peers { permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } }
I20260926 21:00:23.821161 23196 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:23.821134 23204 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } peers { permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } } }
I20260926 21:00:23.821252 23204 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:00:23.821761 23020 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "412c9a44e7974194a3a9b2fb475da533"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37126
I20260926 21:00:23.821815 23046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } peers { permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } }
I20260926 21:00:23.821938 23046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412c9a44e7974194a3a9b2fb475da533, State: Initialized, Role: FOLLOWER
I20260926 21:00:23.822028 23046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } peers { permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } }
I20260926 21:00:23.822343 23046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:23.822412 23020 raft_consensus.cc:482] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 21:00:23.822469 23020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:23.823452 23020 raft_consensus.cc:504] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } peers { permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } }
W20260926 21:00:23.823648 23216 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:00:23.823689 23216 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:00:23.823704 23020 leader_election.cc:288] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [CANDIDATE]: Term 1 election: Requested vote from peers 004b8632c12344ff8356b09f61498b62 (127.0.98.61:33831), f6f7f9567a314a92a764aa0744cf7562 (127.0.98.60:38151)
I20260926 21:00:23.824364 23170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "412c9a44e7974194a3a9b2fb475da533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f6f7f9567a314a92a764aa0744cf7562"
I20260926 21:00:23.824558 23170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:23.825464 23170 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412c9a44e7974194a3a9b2fb475da533 in term 1.
I20260926 21:00:23.823303 23128 master_runner.cc:186] Error getting master registration for 127.0.98.62:37833: OK, instance_id { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" instance_seqno: 1790456423459674 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 21:00:23.825820 23128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 21:00:23.826072 22997 leader_election.cc:302] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412c9a44e7974194a3a9b2fb475da533, f6f7f9567a314a92a764aa0744cf7562; no voters:
I20260926 21:00:23.826336 23234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } peers { permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } } }
I20260926 21:00:23.826414 23234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:00:23.826469 23234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:00:23.826509 23234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [term 1 LEADER]: Becoming Leader. State: Replica: 412c9a44e7974194a3a9b2fb475da533, State: Running, Role: LEADER
I20260926 21:00:23.826596 23234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } peers { permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } }
I20260926 21:00:23.827054 23234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 412c9a44e7974194a3a9b2fb475da533. Latest consensus state: current_term: 1 leader_uuid: "412c9a44e7974194a3a9b2fb475da533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } peers { permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } } }
I20260926 21:00:23.827140 23234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:23.825812 22978 master_runner.cc:186] Error getting master registration for 127.0.98.61:33831: OK, instance_id { permanent_uuid: "004b8632c12344ff8356b09f61498b62" instance_seqno: 1790456423627273 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 21:00:23.828218 22978 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 21:00:23.824460 23228 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:00:23.828280 23228 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:00:23.828341 23236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:00:23.828568 23236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:00:23.830082 23170 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562 [term 1 FOLLOWER]: Refusing update from remote peer 412c9a44e7974194a3a9b2fb475da533: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:00:23.830410 23234 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:00:23.831709 23204 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 412c9a44e7974194a3a9b2fb475da533. Latest consensus state: current_term: 1 leader_uuid: "412c9a44e7974194a3a9b2fb475da533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } peers { permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } } }
I20260926 21:00:23.831812 23204 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:00:23.846657 23234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "412c9a44e7974194a3a9b2fb475da533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } peers { permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } } }
I20260926 21:00:23.846797 23234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:23.846869 23237 mvcc.cc:205] Tried to move back new op lower bound from 7333709512005574656 to 7333709511994658816. Current Snapshot: MvccSnapshot[applied={T|T < 7333709512005574656}]
I20260926 21:00:23.847028 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.837475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37140 (local address 127.0.98.62:37833)
0926 21:00:23.837531 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.837533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.837833 (+ 300us) server_negotiation.cc:407] Connection header received
0926 21:00:23.837946 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.837947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.837959 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.838038 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.838960 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.839752 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.840895 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.841107 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.841193 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.842862 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:23.842864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:23.842865 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:23.842873 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:23.845004 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.845491 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.845493 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.845493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.845582 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.845909 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.845910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.845910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.846083 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:23.846182 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:23.846569 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 21:00:23.846656 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 21:00:23.847502 23204 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "412c9a44e7974194a3a9b2fb475da533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } peers { permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } } }
I20260926 21:00:23.847616 23204 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:00:23.847927 23239 mvcc.cc:205] Tried to move back new op lower bound from 7333709512005574656 to 7333709511994658816. Current Snapshot: MvccSnapshot[applied={T|T < 7333709512005574656}]
I20260926 21:00:23.849143 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.830203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60482 (local address 127.0.98.61:33831)
0926 21:00:23.830248 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.830251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.832593 (+ 2342us) server_negotiation.cc:407] Connection header received
0926 21:00:23.832677 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.832678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.832691 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.832754 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.841436 (+ 8682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.842171 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.843224 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.844043 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.844093 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.845610 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:23.845612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:23.845612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:23.845621 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:23.847501 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.847911 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.847913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.847913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.847976 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.848345 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.848347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.848347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.848526 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:23.848626 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:23.848968 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 21:00:23.849055 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 21:00:23.850689 23236 catalog_manager.cc:1347] Generated new cluster ID: 0bad7f19280a44eba7d1740ec62f7ecd
I20260926 21:00:23.850812 23236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:00:23.863420 23121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } attempt: 3
Sep 26 21:00:23 dist-test-slave-fl7g krb5kdc[22965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 21:00:23.868860 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.837711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.0.98.60:38151)
0926 21:00:23.837765 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.837767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.848493 (+ 10726us) server_negotiation.cc:407] Connection header received
0926 21:00:23.848591 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.848592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.848604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.848634 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.849496 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.850248 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.853406 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.853706 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.853765 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.865495 (+ 11730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:23.865497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:23.865497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:23.865505 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:23.867226 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.867652 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.867653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.867654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.867710 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.867935 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.867936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.867936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.868076 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:23.868182 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:23.868657 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 21:00:23.868746 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 21:00:23.872118 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.863815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38151 (local address 127.0.0.1:41160)
0926 21:00:23.863886 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 21:00:23.863898 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 21:00:23.863941 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:00:23.864102 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:00:23.864102 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:00:23.864298 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:00:23.864349 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:23.864351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.865463 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.865464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:23.866304 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:00:23.866307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.866893 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.866893 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:00:23.866955 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.867390 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:00:23.867395 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:00:23.869098 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:00:23.871141 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:23.871142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:23.871143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:23.871355 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:23.871533 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:00:23.871533 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:00:23.871534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:00:23.871581 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:00:23.871816 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:00:23.871817 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:00:23.871863 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 21:00:23.871920 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 21:00:23.872015 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 21:00:23.872761 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:23.863896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.0.98.60:38151)
0926 21:00:23.863949 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.863951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.863961 (+ 10us) server_negotiation.cc:407] Connection header received
0926 21:00:23.863994 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.863995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.864008 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.864096 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.864547 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.865374 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.866411 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:23.866763 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:23.866824 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:23.869432 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:00:23.869434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:00:23.869434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:00:23.869441 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:00:23.871048 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.871428 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.871429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.871429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.871477 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:00:23.871632 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:00:23.871633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:00:23.871633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:00:23.871744 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:00:23.871795 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:23.872591 (+ 796us) server_negotiation.cc:299] Negotiation successful
0926 21:00:23.872670 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 21:00:23.874382 23121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62: Bootstrap starting.
I20260926 21:00:23.875216 23121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62: Neither blocks nor log segments found. Creating new log.
I20260926 21:00:23.875588 23121 log.cc:824] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62: Log is configured to *not* fsync() on all Append() calls
I20260926 21:00:23.877625 23121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62: No bootstrap required, opened a new log
I20260926 21:00:23.879747 23121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } peers { permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } }
I20260926 21:00:23.879911 23121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004b8632c12344ff8356b09f61498b62, State: Initialized, Role: FOLLOWER
I20260926 21:00:23.879997 23121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } peers { permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } }
I20260926 21:00:23.880364 23121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:00:23.880448 23094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:00:23.881477 23094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62 [term 1 FOLLOWER]: Refusing update from remote peer 412c9a44e7974194a3a9b2fb475da533: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:00:23.881843 23212 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:00:23.882471 23236 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:00:23.883257 23236 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:00:23.884514 23246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } peers { permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } } }
I20260926 21:00:23.884783 23095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "412c9a44e7974194a3a9b2fb475da533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "004b8632c12344ff8356b09f61498b62"
W20260926 21:00:23.888382 23261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:00:23.888487 23261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:00:23.888778 23250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 412c9a44e7974194a3a9b2fb475da533. Latest consensus state: current_term: 1 leader_uuid: "412c9a44e7974194a3a9b2fb475da533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } peers { permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } } }
I20260926 21:00:23.888908 23250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:00:23.888964 23250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "412c9a44e7974194a3a9b2fb475da533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } peers { permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } } }
I20260926 21:00:23.889025 23250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:00:23.889662 23095 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 412c9a44e7974194a3a9b2fb475da533 for term 1 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260926 21:00:23.887331 23246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:00:23.891113 23248 mvcc.cc:205] Tried to move back new op lower bound from 7333709512200241152 to 7333709511994658816. Current Snapshot: MvccSnapshot[applied={T|T < 7333709512005574656}]
I20260926 21:00:23.899199 23234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "412c9a44e7974194a3a9b2fb475da533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412c9a44e7974194a3a9b2fb475da533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } peers { permanent_uuid: "004b8632c12344ff8356b09f61498b62" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "f6f7f9567a314a92a764aa0744cf7562" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38151 } } }
I20260926 21:00:23.899340 23234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533 [sys.catalog]: This master's current role is: LEADER
I20260926 21:00:23.900609 23236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 412c9a44e7974194a3a9b2fb475da533: Generated new TSK 0
I20260926 21:00:23.900923 23236 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 21:00:23.930717 23125 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:23.928820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60488 (local address 127.0.98.61:33831)
0926 21:00:23.928873 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.928876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.929963 (+ 1087us) server_negotiation.cc:407] Connection header received
0926 21:00:23.930078 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.930079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.930092 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.930124 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.930637 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60488: BlockingRecv error: recv got EOF from 127.0.0.1:60488 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 21:00:23.931555 23049 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:23.929124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37150 (local address 127.0.98.62:37833)
0926 21:00:23.929165 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.929167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.930994 (+ 1827us) server_negotiation.cc:407] Connection header received
0926 21:00:23.931104 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.931105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.931116 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.931148 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.931494 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37150: BlockingRecv error: recv got EOF from 127.0.0.1:37150 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 21:00:23.932318 23201 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:23.929664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.0.98.60:38151)
0926 21:00:23.929705 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.929707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.931789 (+ 2082us) server_negotiation.cc:407] Connection header received
0926 21:00:23.931892 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.931893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.931905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.931935 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.932244 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41170: BlockingRecv error: recv got EOF from 127.0.0.1:41170 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 21:00:23.937800 23125 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:23.936722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60504 (local address 127.0.98.61:33831)
0926 21:00:23.936770 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.936772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.937129 (+ 357us) server_negotiation.cc:407] Connection header received
0926 21:00:23.937273 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.937275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.937297 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.937330 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.937739 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60504: BlockingRecv error: recv got EOF from 127.0.0.1:60504 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 21:00:23.941604 23125 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:23.940609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.0.98.61:33831)
0926 21:00:23.940653 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.940655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.940924 (+ 269us) server_negotiation.cc:407] Connection header received
0926 21:00:23.941082 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.941083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.941095 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.941127 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.941538 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60508: BlockingRecv error: recv got EOF from 127.0.0.1:60508 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 21:00:23.945969 23125 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:23.945032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60518 (local address 127.0.98.61:33831)
0926 21:00:23.945088 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.945090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.945384 (+ 294us) server_negotiation.cc:407] Connection header received
0926 21:00:23.945497 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.945498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.945510 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.945540 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.945912 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60518: BlockingRecv error: recv got EOF from 127.0.0.1:60518 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 21:00:23.952526 23125 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:23.951476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.0.98.61:33831)
0926 21:00:23.951523 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.951526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.951785 (+ 259us) server_negotiation.cc:407] Connection header received
0926 21:00:23.951901 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.951902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.951915 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.951947 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.952375 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60532: BlockingRecv error: recv got EOF from 127.0.0.1:60532 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 21:00:23.962997 23125 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:23.962002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60548 (local address 127.0.98.61:33831)
0926 21:00:23.962048 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.962050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.962319 (+ 269us) server_negotiation.cc:407] Connection header received
0926 21:00:23.962446 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.962447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.962476 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.962508 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.962935 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60548: BlockingRecv error: recv got EOF from 127.0.0.1:60548 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 21:00:23.981501 23125 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:23.980525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.0.98.61:33831)
0926 21:00:23.980572 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:23.980574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:23.980872 (+ 298us) server_negotiation.cc:407] Connection header received
0926 21:00:23.980983 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:23.980984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:23.980998 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:23.981029 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:23.981439 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60552: BlockingRecv error: recv got EOF from 127.0.0.1:60552 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 21:00:24.015646 23125 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:24.014943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.0.98.61:33831)
0926 21:00:24.014989 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:24.014991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:24.015103 (+ 112us) server_negotiation.cc:407] Connection header received
0926 21:00:24.015178 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:24.015179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:24.015191 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:24.015241 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:24.015584 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60562: BlockingRecv error: recv got EOF from 127.0.0.1:60562 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 21:00:24.082158 23125 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:24.081242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.0.98.61:33831)
0926 21:00:24.081314 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:24.081317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:24.081443 (+ 126us) server_negotiation.cc:407] Connection header received
0926 21:00:24.081537 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:24.081538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:24.081551 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:24.081634 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:24.082053 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60566: BlockingRecv error: recv got EOF from 127.0.0.1:60566 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 21:00:24.212379 23125 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:24.211500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.0.98.61:33831)
0926 21:00:24.211581 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:24.211584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:24.211697 (+ 113us) server_negotiation.cc:407] Connection header received
0926 21:00:24.211792 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:24.211794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:24.211809 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:24.211890 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:24.212267 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60574: BlockingRecv error: recv got EOF from 127.0.0.1:60574 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 21:00:24.470589 23125 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:00:24.469798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.0.98.61:33831)
0926 21:00:24.469873 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:24.469875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:24.469936 (+ 61us) server_negotiation.cc:407] Connection header received
0926 21:00:24.469997 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:24.469998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:24.470008 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:24.470073 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:00:24.470495 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60590: BlockingRecv error: recv got EOF from 127.0.0.1:60590 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 21:00:24.824204 23216 catalog_manager.cc:1259] Loaded cluster ID: 0bad7f19280a44eba7d1740ec62f7ecd
I20260926 21:00:24.824327 23216 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562: loading cluster ID for follower catalog manager: success
I20260926 21:00:24.825305 23216 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562: acquiring CA information for follower catalog manager: success
I20260926 21:00:24.825649 23216 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f6f7f9567a314a92a764aa0744cf7562: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 21:00:24.889034 23261 catalog_manager.cc:1259] Loaded cluster ID: 0bad7f19280a44eba7d1740ec62f7ecd
I20260926 21:00:24.889176 23261 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62: loading cluster ID for follower catalog manager: success
I20260926 21:00:24.890095 23261 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62: acquiring CA information for follower catalog manager: success
I20260926 21:00:24.890470 23261 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 004b8632c12344ff8356b09f61498b62: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 21:00:24.986429 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:24.983772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.0.98.61:33831)
0926 21:00:24.983869 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:24.983871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:24.983882 (+ 11us) server_negotiation.cc:407] Connection header received
0926 21:00:24.984025 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:24.984027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:24.984087 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:24.984189 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:00:24.984336 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:24.984926 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:24.985668 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:24.985902 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:24.985961 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:24.986129 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:00:24.986245 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:00:24.986324 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:24.986342 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 21:00:24.986361 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":52,"threads_started":1}
I20260926 21:00:24.991158 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:24.988432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.0.98.60:38151)
0926 21:00:24.988558 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:24.988560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:24.988568 (+ 8us) server_negotiation.cc:407] Connection header received
0926 21:00:24.988605 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:24.988606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:24.988645 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:24.988732 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:00:24.988870 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:24.989367 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:24.990158 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:24.990497 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:24.990563 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:24.990831 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:00:24.990952 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:00:24.991027 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:24.991064 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 21:00:24.991087 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260926 21:00:24.996327 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:24.993190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60608 (local address 127.0.98.61:33831)
0926 21:00:24.993393 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:24.993395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:24.993536 (+ 141us) server_negotiation.cc:407] Connection header received
0926 21:00:24.993595 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:24.993596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:24.993607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:24.993677 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:00:24.993865 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:24.994463 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:24.995484 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:24.995796 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:24.995861 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:24.996006 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:00:24.996040 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:00:24.996089 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:24.996192 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 21:00:24.996218 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260926 21:00:24.996496 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:24.993447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41190 (local address 127.0.98.60:38151)
0926 21:00:24.993522 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:24.993525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:24.993798 (+ 273us) server_negotiation.cc:407] Connection header received
0926 21:00:24.993834 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:24.993835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:24.993848 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:24.993901 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:00:24.993994 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:24.994730 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:24.995716 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:24.996021 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:24.996085 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:24.996208 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:00:24.996240 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:00:24.996290 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:24.996354 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 21:00:24.996388 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 21:00:24.997241 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:00:24.993245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.0.98.62:37833)
0926 21:00:24.993393 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 21:00:24.993395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 21:00:24.993659 (+ 264us) server_negotiation.cc:407] Connection header received
0926 21:00:24.993737 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:00:24.993738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:00:24.993836 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:00:24.993920 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:00:24.994093 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:24.994846 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:24.995764 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:00:24.996080 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:00:24.996140 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:00:24.996376 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:00:24.996969 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:00:24.997045 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 21:00:24.997096 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 21:00:24.997121 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":104,"threads_started":1}
I20260926 21:00:24.999029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 22978
I20260926 21:00:25.004715 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 23053
I20260926 21:00:25.009943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVxazDW/build/release/bin/kudu with pid 23128
2026-09-26T21:00:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2832 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5654 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (274218 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST